63 questions · Observability Concepts topic · All types, answers revealed
What is the primary benefit of using a pull-based model for monitoring?
If the scrape fails, Prometheus immediately knows the target is down.
Why this answer
The pull model allows the monitoring server to detect when a target is down and prevents the target from overwhelming the monitoring system.
What is the purpose of 'labels' in Prometheus?
Metric name + labels = unique time series.
Why this answer
Labels provide metadata to metrics, allowing for multi-dimensional querying and filtering (e.g., job='web', instance='1.2.3.4').
Which of these is NOT a Golden Signal?
Storage is an infrastructure concern, not a service signal.
Why this answer
The Four Golden Signals are Latency, Traffic, Errors, and Saturation. 'Storage' is not one of them.
You are deploying Prometheus in a cluster with highly dynamic ephemeral pods. Which concern is the most significant regarding cardinality?
Unique label combinations create new time series, consuming RAM.
Why this answer
High cardinality arises from labels with high-variance values, which can lead to memory exhaustion in the Prometheus TSDB.
A system is experiencing high latency. You need to identify if the latency is caused by external dependencies. Which of the Four Golden Signals is most relevant for this discovery?
Latency measures time spent per request.
Why this answer
Latency measures the time it takes to service a request, which directly indicates performance issues in dependencies.
When designing a monitoring strategy, why should you avoid monitoring 'everything'?
Efficient monitoring focuses on relevant signals.
Why this answer
Excessive metric collection leads to high storage costs, increased query latency, and signal-to-noise ratio degradation (alert fatigue).
Which metric type would best capture the distribution of request latencies?
Histograms provide bucketed distributions.
Why this answer
A Histogram samples observations (usually request durations) and counts them in configurable buckets, allowing for percentiles.
You are monitoring a legacy batch processing job that runs for only 30 seconds once every 24 hours. Which Prometheus component is necessary to collect these metrics?
Pushgateway is designed to temporarily store metrics from short-lived batch jobs until Prometheus scrapes it.
Why this answer
Since the job is short-lived, Prometheus's standard scrape interval will likely miss it. The Pushgateway allows the job to push metrics before exiting.
What is the primary function of the 'scrape_timeout' configuration?
It is the deadline for the scrape response.
Why this answer
It defines how long Prometheus waits for a response from the target before considering the scrape a failure.
Which THREE of these represent the Four Golden Signals?
A Golden Signal.
Why this answer
Latency, Traffic, Errors, and Saturation are the set.
Which of these is considered a 'counter' in Prometheus?
This is a classic counter.
Why this answer
A counter is a cumulative metric that only ever goes up, used for things like request totals.
Which TWO are true about Prometheus alerting rules?
Rules are PromQL expressions.
Why this answer
Alerts are evaluated on the server, and they use PromQL for condition checking.
You are seeing 'Too many active series' in your Prometheus logs. Which action should be your first priority to address this?
This helps isolate the source of high cardinality.
Why this answer
Identifying which labels are causing the high cardinality is the standard first step to prevent OOM (Out Of Memory) crashes.
Why might you use an 'exporter' in Prometheus?
Exporters perform the necessary translation.
Why this answer
Exporters are used to convert metrics from systems that don't natively expose Prometheus format into something Prometheus can scrape.
A legacy batch job runs once per day and finishes in seconds. Why is a standard Prometheus pull model inadequate here?
Pulling requires the target to be available when the scrape occurs.
Why this answer
Prometheus pull intervals are typically set to seconds or minutes; a job that finishes quickly might never be 'up' when Prometheus scrapes it.
Which TWO of the following are primary components of the Prometheus monitoring architecture?
The core alert handler.
Why this answer
Prometheus server and Alertmanager are central components; Grafana and exporters are common but auxiliary or distinct tools.
Which of the following describes the 'pull' model's benefit regarding service discovery?
Prometheus finds the targets.
Why this answer
Prometheus can dynamically discover targets (via Kubernetes API, Consul, etc.) and automatically start scraping, removing the need to register every new pod.
When configuring Service Discovery in Kubernetes, why might you use relabeling?
Relabeling is the primary tool to strip labels to save TSDB memory.
Why this answer
Relabeling is used to modify, drop, or keep metrics based on label values, which is essential for managing cardinality and filtering unwanted data.
Which THREE of the following are considered part of the 'Four Golden Signals' of monitoring?
A core golden signal.
Why this answer
The four golden signals are Latency, Traffic, Errors, and Saturation.
A dashboard shows a latency spike that doesn't correlate with CPU usage. What should you investigate next?
These are common non-CPU bottlenecks.
Why this answer
If CPU is not the bottleneck, the issue is likely I/O saturation, network contention, or a downstream dependency delay.
What is the purpose of a Gauge metric?
Gauges track instantaneous state.
Why this answer
A gauge represents a single numerical value that can arbitrarily go up and down, such as temperature or memory usage.
Which monitoring philosophy is core to the Prometheus architecture?
Prometheus pulls metrics from targets at controlled intervals.
Why this answer
Prometheus favors the pull model, where the server controls the scrape rate and frequency, providing better control over the load on the monitoring system.
What is the primary drawback of using the Pushgateway for long-running services?
Prometheus won't know if the source process is actually dead.
Why this answer
Using the Pushgateway removes the 'up' status check that Prometheus performs on targets, and it can become a bottleneck or single point of failure.
What is the primary goal of observability?
This is the definition of observability.
Why this answer
Observability aims to answer 'why' a system is in its current state by understanding its internal state through outputs (metrics, logs, traces).
You are designing a monitoring architecture for an ephemeral, auto-scaling microservices environment. Why is the Prometheus 'pull' model generally preferred over a 'push' model in this scenario?
Prometheus handles the discovery process, ensuring new targets are scraped automatically without client-side configuration.
Why this answer
Pull models allow the monitoring system to handle service discovery and target management centrally, preventing issues with configuration drift when instances are added or removed dynamically.
When evaluating the 'Traffic' golden signal, what is the best metric to watch for a web service?
This measures traffic volume.
Why this answer
Request rate (requests per second) is the standard way to measure traffic volume.
Your team is defining Service Level Objectives (SLOs) for a web application. Which metric is most indicative of the 'Traffic' golden signal?
Request rate is the standard definition of traffic in the golden signals framework.
Why this answer
Traffic is a measure of demand on the system, typically represented by request rate per second.
How does the 'stale marker' work in Prometheus?
This prevents 'trailing' lines on graphs.
Why this answer
When a target stops responding, Prometheus writes a stale marker to the TSDB so that graphs stop at the last known value rather than showing an infinite line or incorrect data.
Which tool would you pair with Prometheus to visualize the collected data?
Grafana provides the UI/dashboards.
Why this answer
Grafana is the standard visualization tool that integrates deeply with Prometheus as a data source.
What is the primary difference between a metric and a log?
This is the classic distinction between the two.
Why this answer
Metrics are aggregated, numeric data points over time, while logs are timestamped records of specific events.
Which component is strictly required to monitor batch jobs in a Prometheus ecosystem?
Pushgateway acts as a metrics buffer for transient jobs.
Why this answer
The Pushgateway is specifically designed to allow ephemeral batch jobs to push metrics, which Prometheus then pulls.
Which golden signal would you monitor to detect if a specific API endpoint is returning 404s?
Error rates are the primary source for identifying 4xx/5xx issues.
Why this answer
The 'Errors' signal tracks the rate of failed requests, including HTTP 4xx and 5xx codes.
A team is transitioning from a traditional logging system to a metrics-based monitoring approach. Which category of observability data should the team prioritize to specifically identify the 'Latency' golden signal?
Metrics are the standard for measuring the four golden signals, including latency.
Why this answer
Latency measures the time it takes to service a request, which is best captured via time-series metrics rather than unstructured logs or traces.
Which TWO are true about the 'Summary' metric type?
This is their key feature.
Why this answer
Summaries are pre-calculated at the client side and provide quantiles.
Which TWO of these are true about the 'Pull' model?
That is the definition of pull.
Why this answer
Prometheus controls the scrape and can detect target downtime.
Which THREE items are standard techniques to reduce metric cardinality?
Reduces the number of individual series.
Why this answer
Dropping labels, aggregating metrics, and limiting metric frequency are standard ways to reduce cardinality.
You need to measure the P99 latency of your service. Which PromQL function do you use on a Histogram?
This is the correct function for quantiles.
Why this answer
The 'histogram_quantile' function is required to calculate quantiles from bucketed histogram data.
Which TWO of the following are examples of metric data?
Gauge metric.
Why this answer
CPU percentage and total request count are classic metrics.
Which component is responsible for executing queries written in PromQL?
Prometheus parses and executes the query.
Why this answer
The Prometheus server itself contains the query engine that processes PromQL expressions.
Which TWO of these are potential side effects of high-cardinality metrics?
Each series takes memory.
Why this answer
Memory exhaustion (OOM) and slow query performance are the direct results of high cardinality.
An application is emitting logs that contain '5xx' status codes when a service is unavailable. If you want to create an alert that triggers based on the rate of these errors, what is the best practice for observability?
Converting log events to metrics provides a reliable, performant signal for alerting.
Why this answer
Logs are useful for root cause analysis, but for alerting on rates, you should convert the error occurrence into a metric increment, as metrics are more efficient for long-term calculation and alerting.
Why are 'logs' essential for debugging, even with perfect metrics?
Logs allow drill-down into specific failures.
Why this answer
Logs provide the granular, detailed context (like specific error messages or stack traces) that metrics cannot provide.
A developer asks why they should use Distributed Traces instead of simply using high-cardinality metrics to debug a slow request. What is the primary advantage of traces?
Traces follow the lifecycle of a request, revealing exactly where bottlenecks exist in complex architectures.
Why this answer
Distributed traces provide the full execution path and context of a single request across multiple services, which metrics cannot do.
Which THREE factors contribute to a 'high cardinality' problem in a Prometheus environment?
These values are unique per user, leading to massive series growth.
Why this answer
High cardinality is caused by labels with many unique values, like timestamps, user IDs, or un-normalized request paths.
In the context of the Four Golden Signals, what does 'Saturation' measure?
Queued work indicates resource contention.
Why this answer
Saturation measures how 'full' a service is, typically defined by identifying the most constrained resource (e.g., CPU, RAM, or disk IO).
You notice that your Prometheus instance memory usage is growing linearly despite constant traffic. What is the most likely cause?
High label variance causes memory inflation.
Why this answer
If cardinality (unique label combinations) increases over time, the number of active time series keeps growing, increasing RAM usage.
Which TWO statements accurately describe the differences between logs and metrics?
Metrics are highly efficient for time-series aggregation.
Why this answer
Metrics are numerical aggregations (efficient), whereas logs are event-based records (detailed).
Which TWO are common causes of high-cardinality in Kubernetes environments?
These are infinite in number.
Why this answer
Using pod names or unique request IDs as labels creates high cardinality.
What is 'Monitoring Philosophy' regarding alerts?
Alerting on user-facing symptoms is best practice.
Why this answer
Effective alerting should be actionable, low-noise, and focus on symptoms (what the user sees) rather than causes.
Why does the 'rate()' function in PromQL behave unexpectedly on counter resets?
Prometheus handles monotonic counter resets automatically.
Why this answer
A counter reset (going to 0) would produce a massive negative number if not handled. 'rate()' automatically accounts for this by assuming a reset occurred.
Which THREE features are provided by Grafana when used with Prometheus?
Primary function.
Why this answer
Visualization, dashboard templating, and alert integration are core Grafana features.
Which THREE actions are commonly performed during the 'relabeling' phase in Prometheus?
Used for normalization.
Why this answer
Replacing labels, dropping series, and mapping labels are standard relabeling actions.
Which THREE of these are common Prometheus data sources?
Common for fixed targets.
Why this answer
Kubernetes service discovery, static file configs, and Consul are standard discovery methods.
If you have a metric `http_requests_total` and want the increase over 5 minutes, what is the best PromQL syntax?
This returns the count increase.
Why this answer
The `increase()` function is specifically designed to calculate the increase of counter values over a specified range.
Which TWO are common types of Prometheus exporters?
Standard host monitoring.
Why this answer
Node Exporter and Blackbox Exporter are the most common official ones.
An engineer observes a massive spike in Prometheus memory usage after introducing a new custom label that includes a unique user ID. Which observability concept is being violated?
Including highly unique values like user IDs in labels creates excessive time series, causing high memory usage.
Why this answer
High cardinality occurs when a metric label has an unbounded number of unique values, which causes Prometheus to create a unique time series for every combination, leading to memory exhaustion.
A developer adds a 'user_id' label to an HTTP request counter. What is the operational risk?
Each user_id will create a separate time series.
Why this answer
Adding a high-cardinality dimension like 'user_id' can explode the number of time series created, crashing the TSDB.
Which THREE of these are recommended practices for an observability strategy?
Logs provide the 'why'.
Why this answer
Alerting on symptoms, monitoring the Golden Signals, and keeping logs for detailed debugging are best practices.
What is the benefit of the 'service discovery' feature in Prometheus?
Auto-scaling environments require dynamic discovery.
Why this answer
It automatically updates the list of targets to scrape, eliminating manual updates when infrastructure scales.
Why are traces considered distinct from metrics in observability?
This is the fundamental definition of distributed tracing.
Why this answer
Traces provide context on a single request's path through a distributed system, whereas metrics provide aggregated health status.
An SRE team is transitioning from traditional logs to metrics for performance monitoring. Which scenario best justifies using metrics over logs?
Metrics are designed for calculating rates and aggregates over time.
Why this answer
Metrics are numerical representations of data measured over time, making them efficient for time-series analysis and alerting, whereas logs are better for debugging specific events.
What is the primary role of the 'Alertmanager'?
Alertmanager handles the alert lifecycle.
Why this answer
Alertmanager is responsible for grouping, silencing, and routing alerts to external systems like PagerDuty or email.
Which THREE are true regarding the Prometheus TSDB?
Crucial for label filtering.
Why this answer
It uses blocks, WAL, and index files, and it is optimized for time-series data.
Ready to test yourself?
Try a timed practice session using only Observability Concepts questions.