DEA-C01 Data Ingestion and Transformation Practice Test 3 — 15 Questions

Question 1

A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data is then consumed by an AWS Lambda function that transforms each record and writes it to Amazon S3. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors when writing to S3. The team has already increased the Lambda function's memory and timeout. Which action should the team take to resolve the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Implement retries with exponential backoff in the Lambda function for S3 put operations.. The 'ProvisionedThroughputExceededException' error indicates that the Lambda function is being throttled by S3 due to exceeding the bucket's request rate limits. Implementing retries with exponential backoff in the Lambda function for S3 put operations is the correct solution because it allows the function to gracefully handle transient throttling errors by waiting progressively longer between retries, which aligns with AWS's guidance for managing S3 request rate limits.

Answer

Use S3 Batch Operations to write data in batches.

Answer

Increase the number of shards in the Kinesis data stream.

Answer

Enable S3 Transfer Acceleration on the destination bucket.

Question 2

An e-commerce company uses AWS Glue to run ETL jobs that transform clickstream data from Amazon S3. The job reads Parquet files, performs aggregations, and writes the results to Amazon Redshift. The job runs successfully but takes longer than expected. The data volume is increasing. Which design change would MOST improve the job's performance?

Accepted Answer

Increase the number of Glue worker nodes (DPUs) for the job.. Increasing the number of Glue worker nodes (DPUs) directly scales the distributed processing capacity of the ETL job, allowing it to process larger volumes of Parquet data in parallel. This is the most straightforward way to reduce execution time when data volume is growing, as AWS Glue automatically partitions the workload across the additional workers.

Answer

Write the aggregated results to a single large file instead of multiple partitions.

Answer

Convert the Parquet files to CSV to simplify the schema.

Answer

Replace the Redshift target with Amazon Redshift Spectrum.

Question 3

A data engineer needs to ingest JSON data from an on-premises relational database into Amazon S3 every hour. Which AWS service should be used to set up a scheduled, incremental data transfer?

Accepted Answer

AWS Database Migration Service (DMS) with S3 as target.. AWS DMS is purpose-built for migrating databases to AWS targets, including Amazon S3. It supports ongoing replication (change data capture) and scheduled full-load tasks, making it ideal for hourly incremental transfers from an on-premises relational database to S3 without custom scripting.

Answer

Amazon S3 Transfer Acceleration with a cron job.

Answer

AWS Glue with a JDBC connection and a scheduled crawler.

Answer

Amazon Kinesis Data Firehose with a database source.

Question 4

A company is using AWS Glue to process data from Amazon S3. The Glue job reads CSV files and writes Parquet files to a different S3 bucket. The job occasionally fails with 'java.lang.OutOfMemoryError: Java heap space'. The data size varies. Which change should the engineer make to avoid this error?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'java.lang.OutOfMemoryError: Java heap space' error in AWS Glue indicates that the Spark executors ran out of memory while processing the data. Increasing the number of DPUs (Data Processing Units) allocated to the Glue job increases the total memory available across the cluster, allowing larger datasets to be processed without hitting the heap limit. Each DPU provides 4 vCPUs and 16 GB of memory, so adding more DPUs scales memory linearly.

Answer

Convert the CSV files to JSON format before processing.

Answer

Decrease the Spark shuffle partitions in the job script.

Answer

Increase the job timeout setting.

Question 5

A data engineering team uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. They notice that the application's checkpointing is failing intermittently, causing data reprocessing. The application uses a large state. Which configuration change should the team make to improve checkpoint reliability?

Accepted Answer

Increase the checkpointing interval.. Increasing the checkpointing interval reduces the frequency of checkpoint operations, giving the system more time to complete each checkpoint before the next one starts. This alleviates backpressure and resource contention, which is critical when dealing with large state, as checkpointing large state is I/O and CPU intensive and can fail if intervals are too tight.

Answer

Disable checkpointing to avoid failures.

Answer

Switch the state backend from in-memory to RocksDB.

Answer

Increase the parallelism of the application.

Question 6

A data engineer is designing a serverless data ingestion pipeline that uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The data must be transformed using AWS Lambda before being written to S3. Which two steps are required to enable this transformation? (Select TWO.)

Accepted Answer

Configure a Lambda function as a data transformation source in the Firehose delivery stream.. Amazon Kinesis Data Firehose can be configured to invoke a Lambda function as a data transformation source. This allows Firehose to pass incoming records to the Lambda function, which processes and returns the transformed records before they are delivered to the S3 destination. Option C is correct because the Lambda function must return data in the specific format that Firehose expects, including a record ID, result status, and base64-encoded data, otherwise the transformation will fail.

Answer

Set up an S3 event notification to trigger the Lambda function on object creation.

Answer

Subscribe the Lambda function to the CloudWatch Logs log group for the Firehose stream.

Answer

Have the Lambda function write the transformed data directly to the S3 bucket.

Question 7

A company uses AWS Glue to perform ETL on data stored in Amazon S3. The Glue job reads CSV files, converts them to Parquet, and partitions by date. The job runs daily and processes about 500 GB of data. The team wants to optimize costs and performance. Which three actions should the team take? (Select THREE.)

Accepted Answer

Use column pruning to read only necessary columns in the Glue script.. Column pruning in AWS Glue scripts reduces the amount of data read from Amazon S3 by specifying only the columns needed for the ETL transformation. This minimizes I/O and network overhead, directly lowering costs and improving job performance, especially when processing large CSV files.

Answer

Increase the Spark shuffle partitions to 500.

Answer

Write the output as JSON instead of Parquet to avoid compression overhead.

Question 8

A company uses AWS Glue to process streaming data from Amazon Kinesis Data Streams. The job reads JSON records and writes Parquet to Amazon S3. Recently, the job started failing with 'Out of Memory' errors. Which change is MOST likely to resolve the issue?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'Out of Memory' error in AWS Glue indicates that the job's allocated resources are insufficient for the data volume or processing complexity. Increasing the number of DPUs (Data Processing Units) directly increases the available memory and compute capacity, which is the most straightforward fix for OOM errors in Glue streaming jobs. Option C is correct because it addresses the root cause—resource exhaustion—by scaling the job horizontally.

Answer

Enable compression on the Kinesis stream.

Answer

Change the output format from Parquet to ORC.

Answer

Reduce the streaming batch size in the Glue job configuration.

Question 9

A data engineer is designing a data ingestion pipeline for IoT sensor data. The data arrives as JSON via AWS IoT Core, and must be stored in Amazon S3 in partitioned Parquet format. The pipeline must handle late-arriving data (up to 1 hour) and ensure exactly-once processing. Which combination of services should the engineer use?

Accepted Answer

Amazon Kinesis Data Firehose with data transformation via AWS Lambda, delivering to Amazon S3.. Amazon Kinesis Data Firehose is the correct choice because it can directly ingest streaming data from AWS IoT Core, use a built-in AWS Lambda function to transform JSON to Parquet, and deliver the data to Amazon S3 with automatic partitioning. It also supports buffering and retry logic to handle late-arriving data (up to 1 hour) and provides exactly-once delivery to S3 when configured with the appropriate error handling and idempotent transformations.

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation and Amazon S3.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda for transformation and Amazon S3.

Answer

AWS Glue streaming jobs consuming from Amazon Kinesis Data Streams and writing to Amazon S3.

Question 10

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3. The data volume is about 500 GB initially, with daily incremental updates of 10 GB. The pipeline must minimize operational overhead. Which AWS service should be used for the initial and incremental loads?

Accepted Answer

AWS Database Migration Service (DMS) with change data capture (CDC) to Amazon S3.. AWS DMS with CDC is the correct choice because it supports continuous replication from Oracle to Amazon S3 with minimal overhead. It handles both the initial 500 GB full load and ongoing 10 GB daily increments via change data capture, without requiring custom code or complex pipeline management.

Answer

AWS Glue with a JDBC connection and incremental crawl.

Answer

Amazon Kinesis Data Firehose with a custom producer.

Answer

AWS Data Pipeline with a SQL activity and HiveCopyActivity.

Question 11

A company has a Glue ETL job that reads from an Amazon RDS for MySQL table and writes to Amazon S3. The job runs hourly and processes new records based on a 'last_modified' timestamp column. Recently, the job started missing some records because the timestamp in MySQL is stored with microsecond precision but Glue's job bookmark only tracks second precision. Which solution addresses this issue?

Accepted Answer

Use a job parameter to store the last processed timestamp with millisecond precision and query records greater than that value.. AWS Glue job bookmarks track timestamps with only second precision, so records with microsecond differences within the same second are missed. By using a custom job parameter to store the last processed timestamp with millisecond precision and querying records greater than that value, you bypass Glue's bookmark limitation and capture all new or modified records.

Answer

Increase the job frequency to every 30 minutes.

Answer

Run a full refresh of the table each time instead of incremental.

Answer

Modify the MySQL table to use a DATE data type instead of TIMESTAMP.

Question 12

A data engineer is ingesting CSV files from an Amazon S3 bucket into a Glue Data Catalog table. The files have headers, but some files have extra columns not present in the first file. The engineer wants the Glue crawler to automatically detect the schema. Which crawler configuration option should be used?

Accepted Answer

Configure the crawler to 'Create a single schema for each S3 path' and enable 'Merge tables'.. When CSV files have varying schemas (extra columns), the Glue crawler must be configured to 'Create a single schema for each S3 path' with 'Merge tables' enabled. This configuration instructs the crawler to union the schemas from all files in the S3 path, adding new columns as they appear, rather than creating separate tables for each schema variation.

Answer

Configure the crawler to 'Inherit schema from table' and set the table name.

Answer

Configure the crawler to 'Create a single schema for each S3 path' without enabling 'Merge tables'.

Answer

Configure the crawler to 'Create a single schema for each S3 path' and set 'Each file as a separate table'.

Question 13

A company is building a data lake on Amazon S3. Data arrives from multiple sources in JSON, CSV, and Avro formats. The data must be transformed to Parquet and partitioned by date and source. Which TWO services can perform this transformation with minimal custom code? (Choose TWO.)

Accepted Answer

Amazon EMR with Spark. Both Amazon EMR with Spark and AWS Glue ETL jobs can perform the transformation with minimal custom code. Spark natively supports reading JSON, CSV, and Avro formats and writing Parquet with partitioning by date and source, requiring only a concise PySpark or Scala script. Similarly, AWS Glue provides a managed Spark environment with built-in transforms and crawlers, allowing users to write Spark scripts or use visual ETL jobs with minimal code. The other options either lack native transformation capabilities (Lake Formation, Athena CTAS queries) or are designed for streaming data (Kinesis Data Firehose).

Answer

AWS Lake Formation

Answer

Amazon Athena CTAS queries

Answer

Amazon Kinesis Data Firehose

Question 14

A data engineer is troubleshooting an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but 5% of records are missing after the load. The engineer suspects data consistency issues. Which THREE actions could help diagnose and resolve the problem? (Choose THREE.)

Accepted Answer

Use the Redshift COPY command with a manifest file to load data.. Using the Redshift COPY command with a manifest file ensures that only the exact files listed in the manifest are loaded, eliminating the risk of partial or duplicate reads from S3. This is a common pattern to guarantee data consistency when the Glue job may not reliably track which files have been processed, especially in scenarios with concurrent writes or retries.

Answer

Increase the number of DPUs for the Glue job.

Answer

Use a staging table in Redshift with a transaction to commit.

Question 15

A company uses AWS Glue to process CSV files from an S3 bucket. The job fails intermittently with a 'SchemaDetectionError' for files that have inconsistent column counts. What is the most efficient way to handle this?

Accepted Answer

Define a fixed schema in the Glue job using 'apply_mapping' to map columns.. Defining a fixed schema using the `schema` parameter in the DynamicFrame reader forces Glue to apply that schema to all CSV files. With `apply_mapping`, you can map the actual columns present to the fixed schema, handling inconsistent column counts by ignoring extra columns and filling missing columns with nulls. This avoids schema detection errors without extra preprocessing.

Answer

Use the 'mergeSchema' option when reading the DynamicFrame.

Answer

Convert all CSV files to Parquet format using a separate preprocessing job.

Answer

Set the job to 'ignore' schema mismatches in the job parameters.