CCNA Structured Streaming Questions

Checkpointing records the metadata and offsets of the micro-batches in a persistent store. This allows the streaming query to recover from failures and resume processing exactly where it left off, which is the cornerstone of providing the exactly-once fault-tolerance guarantees expected in enterprise data pipelines.

The checkpoint location persists the streaming query's progress, including offsets and state, so after a failure the query can resume from the last committed offset and avoid reprocessing or data loss. This is fundamental to Structured Streaming's exactly-once semantics when used with a replayable source and idempotent sink.

Exactly-once semantics require that the sink can handle replays without duplicating data. Delta Lake provides transactional writes and idempotent merges, which allow the streaming query to retry a micro-batch without creating duplicates. In this scenario, writing to a Delta table as the sink ensures that even if a batch is reprocessed after failure, the result remains consistent, thus satisfying exactly-once.

In append mode, only new rows that have been added to the result table since the last trigger are output. For a non-aggregated streaming query, this means each new record is emitted once. This matches the typical debugging use case where you want to see incoming data as it arrives. Therefore this statement correctly describes append mode behavior.

The checkpoint location stores the progress information of the streaming query, including which offsets have been processed. On restart, Spark uses this to resume from where it left off, ensuring each record is processed exactly once. Combined with idempotent sinks like Delta Lake, this provides end-to-end exactly-once guarantees. Therefore, setting a checkpoint location is essential.

A checkpoint location records Kafka offsets and commit metadata durably, letting Structured Streaming resume exactly where it stopped after a restart. Without it, the query cannot recover progress, so exactly-once delivery and automatic restart recovery are impossible.

A durable checkpoint location preserves query progress and state across restarts, satisfying the no-reprocessing requirement. The 30-minute watermark allows the engine to accept and incorporate late events up to 30 minutes past the window boundary. The `append` output mode emits a window's final result only after the watermark passes the window end, ensuring each window is emitted exactly once to the Delta sink.

Watermarks define how long the engine waits for late data. Any event whose event-time falls behind the current watermark is considered too late and is dropped to prevent unbounded state accumulation in streaming aggregations and joins.

The checkpoint location records the progress of the streaming query, specifically the offsets of the data that has been processed. This mechanism enables Spark to resume from the exact point of failure, ensuring fault tolerance and preventing data loss or duplication when the streaming application is restarted after an interruption.

Using from_json with a predefined schema is the standard and most performant approach in Structured Streaming. It allows Spark to parse the JSON content efficiently while enforcing schema constraints, ensuring data quality and type safety, which is essential for downstream analytical workloads and reliable streaming pipelines in production.

A checkpoint location is essential for fault tolerance and exactly-once semantics. It stores the progress of the streaming query, including offsets processed and state information. On restart, the query resumes from where it left off, ensuring no data is lost or duplicated. Without a checkpoint, the job cannot recover correctly.

This configuration perfectly matches the requirement by grouping events into 10-minute buckets while allowing a 2-minute buffer for late data. The watermark correctly signals to Spark that state for windows older than 2 minutes from the maximum event time can be safely cleared, optimizing memory usage and ensuring production stability.

Update mode emits only the rows that were updated since the previous trigger, which is exactly the requirement. For windowed aggregations, as new events arrive and counts change, the affected window rows are re-emitted each trigger with their updated values. This provides near-real-time visibility into evolving aggregates without reprinting the entire result set, making it suitable for a debugging console sink where you want to observe changes.

The watermark specifies how long the engine waits for late data. State for a window is kept until the watermark (max event time seen minus 15 minutes) passes the window's end time. Data arriving within the 15-minute delay is still processed and can update the aggregation. After the watermark passes, state is dropped and further late data is ignored. This is the intended behavior for handling late data.

A checkpoint location is required for any Structured Streaming query to track progress and maintain state. It stores metadata about which offsets have been processed, as well as aggregation state. Without it, the query cannot recover from failures and would either fail to start or lose data. Configuring `checkpointLocation` ensures fault tolerance and exactly-once processing when combined with a replayable source and idempotent sink.

Tightening the watermark duration instructs Spark to discard state for old data sooner. This directly reduces the memory footprint of the state store, as the engine no longer needs to track windows that have already 'expired' according to the watermark, which is the most effective way to address memory pressure.

The checkpoint location stores progress information, including which files or offsets have been processed and the state of stateful operations. On restart, the engine reads this metadata to resume exactly where it left off, preventing reprocessing of already-handled files. Without a checkpoint, the query cannot recover its position and will either fail to restart or reprocess from the beginning, so this is essential for resilient incremental processing.

In append mode, records with event times older than the current watermark are considered too late and are dropped. The watermark is at 10:10, and the record's event time is 10:00, which is 10 minutes behind, exceeding the 5-minute delay. Therefore the record is discarded and will not appear in the output. This is the correct behavior for this scenario.

State timeouts enable the engine to remove inactive state entries after a defined period. By using processing or event-time timeouts, you ensure that memory is reclaimed for keys that are no longer active, which is the most effective way to prevent OutOfMemory errors in stateful streaming.

Continuous processing is specifically engineered to provide sub-millisecond latency by avoiding the batch scheduling overhead present in micro-batch processing. By running tasks continuously on the executors, it eliminates the start-stop cycle, making it ideal for extremely latency-sensitive applications that require instantaneous response times to streaming data inputs.

`StreamingQuery.lastProgress` returns a JSON object containing detailed metrics for the most recent micro-batch, including input rows per second, processing rate, batch duration, and more. It is a direct API call on the streaming query object and provides the most immediate and structured access to the query's status. This method is commonly used to programmatically monitor streaming queries.

Append mode outputs only new rows that have been added since the last trigger. Since there are no aggregations, each micro-batch contains only the new data from Kafka. This matches the requirement to see only newly arrived data. Therefore, Append mode is correct.

When a Structured Streaming query reads from a Delta table and starts without a checkpoint, the Delta source processes the entire current snapshot of the table as the first micro-batch, then follows with new changes. This is the default behavior, so the developer does not need to set any special option to read existing data on the first run.

In Structured Streaming, when a watermark is defined on an event-time column, any event whose event-time timestamp is older than the current watermark (max event time seen minus the watermark delay) is considered too late and is dropped from the aggregation. This is the documented behavior: watermarking allows the engine to discard late data to bound state. Since the job uses append mode with a watermark, late events beyond the threshold are excluded from the result set, which is the correct outcome here.

dropDuplicatesWithinWatermark combined with withWatermark limits how long the engine retains keys for deduplication. Once the watermark passes a key's event time beyond the configured delay, the key can be evicted from state, bounding memory usage. This directly addresses unbounded state growth while still deduplicating events that arrive within the allowed lateness window, which matches the requirement to remove duplicates within a reasonable period.

In append mode, windowed aggregations emit a window's result only once the watermark has advanced past the window's end time, guaranteeing no further updates. If the watermark is large or the data stream is short, the watermark may not have passed any window end yet, so no rows are written. This explains why the query processes data but the Delta table remains empty.

To achieve exactly-once semantics when writing to an external database via `foreachBatch`, the writes must be idempotent. The batch ID provided by `foreachBatch` is unique for each micro-batch and can be used to deduplicate writes. For example, you can store the batch ID in the database and check it before applying the batch, or use it as part of an upsert key. Combined with checkpointing, which ensures that the stream resumes from the correct batch after a failure, this provides exactly-once semantics. Without idempotent writes, a failure could cause a batch to be reprocessed, leading to duplicates.

The Kafka Structured Streaming source uses `startingOffsets` to decide where the very first query starts when no checkpoint exists. Its default value is "latest", which means the query begins at the newest offset per partition. Older records already present in the topic are therefore never delivered to the first micro-batch, exactly matching the observed behavior.

In a stream-stream join, the engine maintains state for both sides to match records. Watermarks are used to determine when state can be evicted. For an inner join, state for a record is kept until the watermark passes the record's event time plus the watermark delay, ensuring that late matches can still occur within the allowed lateness.

When no trigger is specified, Structured Streaming uses the default trigger, which processes each micro-batch as soon as the previous one finishes. This provides the lowest latency without a fixed interval. It is the standard behavior for streaming queries that need continuous processing. The developer does not need to set a trigger for this default behavior.

Delta Lake is a first-class streaming source in Spark. It uses the transaction log to track data changes, allowing it to efficiently read new rows as they are added to the table. This is the recommended approach for building robust streaming pipelines on the Databricks Lakehouse platform.

foreachBatch provides the batch identifier, which is monotonically increasing and stable across retries of the same batch. By using it to make the write idempotent, such as merging on a batch column or replacing a partition keyed by batchId, a retried batch overwrites rather than duplicates its prior output. This delivers exactly-once effects at the sink even when the batch function is re-executed.

If `maxOffsetsPerTrigger` is set to 10000 and the Kafka topic receives more than 10000 new records per trigger interval, the query will only process 10000 records per micro-batch. The backlog will grow because the processing rate is capped below the arrival rate. To reduce the backlog, you would need to increase `maxOffsetsPerTrigger` or increase the trigger frequency, provided the cluster can handle the load.