Observability Concepts practice questions

Grafana

Why it fails: Grafana renders dashboards and alerts from data Prometheus already stores; it cannot collect or scrape batch-job metrics itself. The question asks what is strictly required for monitoring, meaning the component that gathers and stores time-series data. Grafana is the right choice when visualising existing Prometheus metrics, not as the collection prerequisite.

Alertmanager

Why it fails: Alertmanager only routes and deduplicates alerts; it plays no part in collecting batch job metrics. It would be correct when notifications on those metrics are required, but scraping and storing them needs only Prometheus plus Pushgateway.

PromQL

Why it fails: PromQL is Prometheus's query language, not a deployable component, so it cannot be a required piece of the monitoring pipeline. It would be correct when writing queries or alerting rules, but batch job collection needs Pushgateway and Prometheus.

Configuring the storage retention policy.

Why it fails: Retention policy is set in the storage configuration block, not in relabeling, which rewrites target and sample labels during scrape or write. Retention configuration is the right action when managing disk usage and data lifetime, not when filtering or transforming labels.

Increasing the memory of the Prometheus instance.

Why it fails: Memory allocation is a deployment or resource-limit setting, not a relabeling action; relabeling only rewrites labels on targets and samples. Raising memory is the right response to OOM kills or heavy cardinality, but it does not belong in the relabeling phase.

The network bandwidth used by the pushgateway.

Why it fails: Pushgateway bandwidth is unrelated to cardinality; cardinality concerns the number of distinct time series, not traffic volume. Pushgateway is tempting because it handles short-lived jobs, and would be the right answer if the question asked how to capture metrics from batch jobs that finish before scraping.

The disk space used by WAL logs.

Why it fails: WAL disk usage scales with sample ingestion and retention, not with the number of distinct label combinations. It is tempting because ephemeral pods do generate churn and storage pressure, and WAL sizing would be the correct concern if the question asked about disk capacity planning or retention.

The number of targets being scraped.

Why it fails: Target count affects scrape load and scheduling, but cardinality is driven by distinct label-value combinations per series, which churning pod names, UIDs and hashes inflate. It is tempting because more targets usually mean more series, yet target quantity alone is not the cardinality mechanism.

CPU throttling.

Why it fails: Linear memory growth with constant traffic points to rising cardinality, such as new label values creating fresh time series, not CPU throttling, which slows processing rather than consuming memory. Throttling is tempting because resource pressure often implicates CPU, but the symptom here is series accumulation.

Network latency.

Why it fails: Network latency affects scrape duration and can cause missed scrapes, but it does not make Prometheus memory grow linearly under constant traffic. It is tempting because latency is a common operational concern, and it would be the right focus when diagnosing slow queries or scrape timeouts, not steady memory growth.

Too many alert rules.

Why it fails: Alert rules are evaluated against already-stored samples and consume negligible memory compared with ingestion; they cannot explain linear growth under constant traffic. Alerting configuration is the correct area when investigating missing or misfiring alerts, but not when diagnosing steadily rising Prometheus memory.

The number of 5xx errors.

Why it fails: Saturation measures how full a constrained resource is, such as queue depth or memory utilisation approaching its limit; 5xx counts belong to the Errors signal. It tempts because error rates are a genuine golden signal, and counting server errors would be correct if the question instead asked what the Errors signal measures.

The speed of the network.

Why it fails: Saturation concerns resource fullness, not throughput or link speed; network speed is a capacity attribute, not a measure of how close a resource is to its limit. It tempts because network performance affects saturation indirectly, and measuring link speed would be right when assessing raw bandwidth capacity rather than resource exhaustion.

The number of successful requests.

Why it fails: Successful request counts describe Traffic, the demand placed on the system, not Saturation, which quantifies resource fullness against a limit. It tempts because request volume drives saturation, and counting successful requests would be correct if the question asked what the Traffic golden signal measures.

The data is too large for memory.

Why it fails: Data volume is irrelevant; a daily job emits a handful of samples. The pull model fails because the job exits within seconds, so scrapes at the interval usually miss the process entirely. The distractor tempts because batch workloads are associated with large datasets, but size is not the mechanism at issue.

Pull model is insecure.

Why it fails: Pull scraping uses the same HTTP transport and TLS options as any other endpoint, so security is not the limiting factor. The real failure is that the short-lived job exits between scrapes, leaving no target to poll. The distractor tempts because unauthenticated /metrics endpoints can be exposed, but that is a configuration choice.

Prometheus cannot scrape batch jobs.

Why it fails: Prometheus can scrape any HTTP endpoint exposing metrics, including batch jobs, so the claim is false. The genuine problem is the job's seconds-long lifetime against a fixed scrape interval, meaning samples are missed. The distractor tempts because batch jobs are assumed unsupported, yet Pushgateway exists precisely to bridge them.

Errors

Why it fails: Errors counts failed requests, so a dependency that responds slowly but successfully produces no error signal at all. High latency with zero errors is exactly the case here. Errors would be the correct signal when the symptom is failed requests, exceptions or non-2xx responses rather than delay.

Saturation

Why it fails: Saturation describes resource utilisation approaching a limit, such as CPU, memory or connection pools, which is internal to the system. A slow external dependency consumes no local resource while waiting. Saturation would be the right signal when latency correlates with resource exhaustion under load.

Traffic

Why it fails: Traffic measures request volume or demand on the system, revealing load patterns rather than where time is spent. External dependency latency appears as elevated duration in the requests that call those services. Traffic would be the right signal when diagnosing capacity-driven demand spikes or unexpected request surges.

It requires manual restarts.

Why it fails: The Pushgateway does not require manual restarts; it runs continuously and accepts pushes over HTTP. Its actual drawback for long-running services is that pushed metrics are not automatically expired and can mask instance failures. Manual restarts would be relevant to a misbehaving exporter process, not the gateway itself.

It lacks metric persistence.

Why it fails: The Pushgateway does persist metrics; it stores pushed samples until they are overwritten or deleted, which is why stale data lingers. Its drawback for long-running services is that it cannot distinguish instances and loses up/down semantics. Persistence is precisely what makes it useful for short-lived batch jobs.

Prometheus cannot query the Pushgateway.

Why it fails: Prometheus does query the Pushgateway; it scrapes the gateway as a target and exposes pushed metrics through the standard query API. The real drawback concerns metrics from long-running services being overwritten or lacking instance-level detail. Querying the Pushgateway is exactly how pushed metrics become visible.

Auditing user access attempts for compliance.

Why it fails: Metrics are aggregated numeric time series and cannot retain per-request identity, so they cannot evidence who accessed what for an audit. Auditing is tempting because metrics show access volume trends, but that scenario would instead justify metrics for capacity or anomaly alerting.

Storing the payload of a failed transaction for later inspection.

Why it fails: Metrics store numeric aggregates, not raw payload content, so a failed transaction's body cannot be reconstructed from them. Storing payloads is tempting because metrics flag failure rates, yet this scenario requires logs, which retain the full request detail for inspection.

Recording the stack trace of a Java application crash.

Why it fails: Metrics suit aggregate, numeric, time-series questions like request rates and latency percentiles; a stack trace is high-cardinality, unstructured text requiring full context, which logs capture and metrics cannot. The appeal is that crashes are performance-relevant, yet diagnosing them needs the detailed event record logs provide.

Metrics are for debugging; logs are for alerting.

Why it fails: Metrics are numeric time-series samples aggregated for dashboards and threshold alerting, while logs are discrete timestamped event records used for debugging detail. Reversing their purposes misstates the actual axis of difference: aggregation versus event detail. Metrics-driven alerting is the correct pattern when you need cheap, continuous threshold detection.

Logs are cheaper to store.

Why it fails: Log storage cost depends on volume, retention and indexing, not on the log-versus-metric distinction; high-cardinality indexed logs are typically costlier per event than aggregated numeric samples. The genuine axis is structure: metrics are pre-aggregated numeric time series, logs are discrete event records. Cost is a deployment choice, not a defining property.

Logs are always real-time.

Why it fails: Real-time delivery is a pipeline property, not a defining trait; logs can be batched or delayed in transit just as metric scrapes occur at fixed intervals. The actual axis is data shape: metrics are numeric time-series samples, logs are discrete timestamped events. Real-time streaming is a configuration choice for either signal type.

Traffic

Why it fails: Traffic counts requests, not response codes, so 404s are invisible in it; you would need a metric labelled by status code. Traffic is genuinely useful for spotting demand shifts or throughput drops across a service, which is why it tempts here.

Saturation

Why it fails: Saturation measures resource pressure such as CPU or queue depth, and a 404 consumes negligible resources, so it cannot reveal the status code returned. Saturation is the right signal when diagnosing capacity exhaustion or overloaded workers.

Latency

Why it fails: Latency measures response time, not HTTP status codes, so 404s remain invisible. It is tempting because latency is a core golden signal for detecting slowdowns and saturation-driven degradation, and would be the right choice when diagnosing slow endpoints or performance regressions rather than error responses.

To encrypt metric traffic.

Why it fails: Relabeling rewrites, drops, or adds target labels during service discovery to control which targets are scraped and how they are labelled; it does not encrypt traffic, which requires TLS configuration on the scrape endpoint. It is tempting because relabeling touches scrape configuration, but its purpose is label and target manipulation.

To increase scrape frequency.

Why it fails: Scrape frequency is set by scrape_interval in the scrape config, not by relabeling, which only rewrites target labels and metadata before scraping. The confusion arises because relabeling sits in the same configuration block, yet it cannot alter timing; it filters or transforms targets, for example dropping unwanted pods.

To store logs in Prometheus.

Why it fails: Relabeling rewrites target labels during service discovery; it never stores logs, which Prometheus does not handle as a log store at all. It is tempting because relabeling manipulates metadata, but its purpose is filtering and reshaping scrape targets, not persisting event data.

Traces are always numeric.

Why it fails: Traces are structured span data carrying timestamps, service names, durations and parent-child relationships, not numeric values. It is tempting because trace durations and counts can be aggregated into metrics, but the trace itself remains event context, which is precisely what distinguishes it from metrics.

Traces are only for frontend applications.

Why it fails: Traces span backend services, databases and queues, not just frontend applications, so restricting them to frontend work misstates their scope. It is tempting because browser instrumentation often generates the first spans, yet distributed tracing's purpose is correlating requests across every component in the path.

Metrics replace traces.

Why it fails: Metrics and traces are complementary signals; metrics aggregate numeric measurements over time, while traces record individual request paths with span context. Neither replaces the other. It is tempting because both feed dashboards and alerting, but replacing traces would discard per-request causality that aggregated metrics cannot reconstruct.

Agent-less log aggregation.

Why it fails: Prometheus scrapes numeric time-series metrics over HTTP; it does not aggregate logs, and its own logs are unstructured. Log aggregation describes agent-based pipelines such as Fluentd or Loki, which would be the right pick when the requirement is centralised searching and parsing of application log lines rather than metric collection.

Push-based data ingestion.

Why it fails: Prometheus pulls metrics by scraping HTTP endpoints on a schedule; push ingestion applies only to short-lived jobs via the Pushgateway. A genuinely push-based design suits Graphite or StatsD, where agents emit metrics to a collector, so this would be correct if targets could not expose a scrapeable endpoint.

Event-driven architecture.

Why it fails: Prometheus is a pull-based time-series database with periodic scraping and rule evaluation, not an event bus reacting to discrete messages. Event-driven architecture describes systems such as Kafka or Fluent Bit pipelines, and would be the right answer if the requirement were reacting to individual events rather than sampling numeric metrics.

It supports higher data throughput.

Why it fails: Pull's benefit is target discovery, health visibility and centralised scrape configuration, not throughput; push handles high-volume, short-lived jobs better. Throughput is tempting because pull spreads scrape load, but the model's real gain is that Prometheus controls scrape timing and detects down targets via failed scrapes.

It is easier to configure for firewalls.

Why it fails: Push is often easier for firewalls.

It avoids network security issues.

Why it fails: Pull still traverses the network and needs reachable endpoints, so it does not avoid security issues; it merely shifts connection direction. It is tempting because scrape targets initiate no inbound connections, which suits segmented or NAT-bound environments, but that is a topology advantage, not the primary benefit of pull-based monitoring.

Switch to a different monitoring tool.

Why it fails: Replacing the monitoring tool abandons the existing metrics and dashboards without reducing cardinality, which stems from high-label combinations in the ingested series. It is tempting when the limit feels insurmountable, but the correct first step is identifying and dropping or aggregating the offending high-cardinality labels.

Disable all alerting rules.

Why it fails: Disabling alerting rules removes notifications but leaves the cardinality explosion untouched, so ingestion memory pressure continues. Alerting rules are for routing symptom-based notifications to responders, and disabling them would be the right move only when noisy alerts themselves are the problem, not series growth.

Increase Prometheus memory limits.

Why it fails: Raising memory limits postpones the out-of-memory crash without reducing the label combinations generating the series, so the underlying cardinality problem persists and recurs. Memory tuning suits steady, predictable growth; here the first priority is identifying and dropping the high-cardinality labels or metrics causing the explosion.

The query speed will increase.

Why it fails: Adding a high-cardinality label such as user_id multiplies time series, increasing memory, storage and query cost, so speed degrades rather than improves. It is tempting because labels enable filtering, which feels like it should speed up targeted queries, but cardinality growth overwhelms that benefit.

Prometheus will reject the metric.

Why it fails: Prometheus accepts the metric; it does not validate label cardinality or reject series at ingestion. The real risk is cardinality explosion degrading performance and storage. It is tempting because invalid metric names or malformed exposition lines are rejected, so label misuse appears similar, but user_id is syntactically valid.

The metric will stop recording.

Why it fails: High-cardinality labels such as user_id create a separate time series per user, exhausting memory and slowing queries, but the counter keeps recording. It is tempting because labels normally add useful dimensions for filtering; with a bounded set of values, such as status codes, labelling is entirely correct.

Temperature of a server room.

Why it fails: Temperature is a gauge: it rises and falls, so it cannot be a counter, which only increases (or resets to zero on restart). Temperature is correctly modelled as a gauge when tracking server-room conditions over time.

Average latency over 5 minutes.

Why it fails: Average latency over 5 minutes is a computed aggregate, not a stored metric type; Prometheus counters are cumulative values that only increase, such as total requests served. Averages belong to gauges, which rise and fall, so this would be the right classification when asked about a value like current memory usage or queue depth.

Current memory usage of a process.

Why it fails: Current memory usage is a gauge: it rises and falls freely, so Prometheus stores its instantaneous value rather than a monotonic total. Counters only increase, or reset to zero on restart, and suit cumulative events such as requests served or errors raised. Gauges are the right choice for metrics like memory, temperature or queue depth.

To track request rates.

Why it fails: Request rates are captured by Counters, which only increase and suit rate() calculations. A Gauge records a value that can rise or fall at any moment, such as current memory usage or queue depth, so it would be the right choice for instantaneous state rather than cumulative counts.

To track job start times.

Why it fails: Job start times are timestamps, typically exported as a Gauge holding the Unix epoch seconds of the last run, but the metric's purpose is representing a current snapshot value, not recording event times. Timestamps for scheduling belong to exporter metadata or logs.

To track request durations.

Why it fails: Request durations are measured with Histograms or Summaries, which bucket or quantile observations for aggregation. A Gauge holds a single instantaneous value that can go up or down, so it cannot capture the distribution of many duration samples needed for latency percentiles.

Logs are always better for everything.

Why it fails: Logs are one telemetry type among metrics, traces and profiles, each answering different questions; no single signal replaces the others. It tempts because logs offer high detail for debugging, and choosing logs alone would be defensible when the requirement is deep forensic investigation of a specific failure rather than broad monitoring.

The network will be saturated.

Why it fails: Collecting extra metrics adds scrape, storage and cardinality load on Prometheus and its targets, not link saturation; network bandwidth is rarely the binding constraint. It tempts because monitoring traffic does traverse the network, and bandwidth exhaustion would be the genuine concern in a high-frequency scrape environment across many remote targets.

Prometheus cannot handle many metrics.

Why it fails: Prometheus scales to millions of active series; the real constraint is cardinality and query cost, not a hard metric ceiling. It tempts because large metric volumes do strain Prometheus, and capacity limits would be the correct answer if the question concerned cardinality explosion or resource sizing rather than monitoring discipline.