DEA-C01 Data Ingestion and Transformation Practice Questions

Explanation: Options A and E are both correct. Option A increases the Spark memory overhead parameter (`spark.executor.memoryOverhead`), which allocates additional off-heap memory for Spark operations. This is a cost-effective tuning approach that can prevent 'Out of Memory' errors without adding more workers or changing job types. Option E changes the worker type from 'G.1x' to 'G.2x', doubling the memory per worker. While this increases per-worker cost, it avoids adding more workers (which would increase cost significantly) and is a targeted fix for memory-intensive workloads. Both options address memory pressure without the high cost of increasing the number of workers (Option C) or the performance loss of switching to Python shell (Option D).

Explanation: AWS Glue streaming ETL, while capable of handling late data via Spark Structured Streaming's watermarking, is not ideal for near-real-time Parquet partitioning to S3 at high throughput; it is more suited for complex transformations with micro-batches. Option B is incorrect: Kinesis Data Analytics tumbling windows process data in memory and output to destinations like S3, but they lack built-in late data handling and Parquet conversion. Option C is correct: Kinesis Data Firehose delivers streaming data to S3 with near-real-time latency, and a Lambda transformation converts it to Parquet. To handle late-arriving data (up to 1 hour), a separate Lambda consumer reads from the Kinesis Data Streams shard iterator to reprocess records that arrive after the Firehose delivery window, ensuring no data loss. Option D is incorrect: S3 Batch Operations are designed for batch processing of existing objects, not for near-real-time handling of late-arriving streaming data.

Explanation: The AWS Glue crawler uses built-in classifiers to infer schema of CSV files. The built-in CSV classifier automatically detects and skips header rows. However, if a custom classifier is configured and it does not have the correct setting to skip headers (e.g., `HasHeader` set to `false` or not defined), the crawler will treat the header row as data. Since the question describes the header being included as data, the most likely cause is that a custom classifier is in use that does not properly skip headers.

Explanation: Amazon Kinesis Data Firehose can deliver data directly to Amazon Redshift via an intermediate Amazon S3 bucket, making option A a factor to consider. Option C is a factor because Kinesis Data Firehose automatically scales its throughput, while Kinesis Data Streams requires manual shard management. Option E is a factor because Kinesis Data Streams supports custom data processing using the Kinesis Client Library (KCL) or AWS Lambda, whereas Firehose only supports Lambda for transformation.

Explanation: AWS Glue job bookmarks can track processed records by using a timestamp column in the source table, enabling incremental processing (Option C). Enabling binary logging (binlog) on RDS MySQL is a necessary step for any change data capture (CDC) implementation, as it ensures all changes are recorded and can be used by AWS DMS or other services to capture changes in real time, which can then be ingested into AWS Glue via S3 or Kinesis. Although Glue does not directly read binlog via JDBC, enabling binlog is a best practice and a prerequisite for CDC from RDS MySQL. Additionally, using a full table scan with a WHERE clause on a timestamp column (Option E) is an efficient way to process only new and modified records when combined with job bookmarks. Option A is incorrect because Glue JDBC connections cannot directly read MySQL binary logs. Option B is unnecessary and adds complexity without direct benefit to the task.