DEA-C01 Data Ingestion and Transformation Practice Test 2 — 15 Questions

Question 1

A data engineering team uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. They notice that the application's checkpointing is failing intermittently, causing data reprocessing. The application uses a large state. Which configuration change should the team make to improve checkpoint reliability?

Accepted Answer

Increase the checkpointing interval.. Increasing the checkpointing interval reduces the frequency of checkpoint operations, giving the system more time to complete each checkpoint before the next one starts. This alleviates backpressure and resource contention, which is critical when dealing with large state, as checkpointing large state is I/O and CPU intensive and can fail if intervals are too tight.

Answer

Disable checkpointing to avoid failures.

Answer

Switch the state backend from in-memory to RocksDB.

Answer

Increase the parallelism of the application.

Question 2

A data engineer is designing a serverless data ingestion pipeline that uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The data must be transformed using AWS Lambda before being written to S3. Which two steps are required to enable this transformation? (Select TWO.)

Accepted Answer

Configure a Lambda function as a data transformation source in the Firehose delivery stream.. Option B is correct because Amazon Kinesis Data Firehose can be configured to invoke a Lambda function as a data transformation source. This allows Firehose to pass incoming records to the Lambda function, which processes and returns the transformed records before they are delivered to the S3 destination. Option C is correct because the Lambda function must return data in the specific format that Firehose expects, including a record ID, result status, and base64-encoded data, otherwise the transformation will fail.

Answer

Set up an S3 event notification to trigger the Lambda function on object creation.

Answer

Subscribe the Lambda function to the CloudWatch Logs log group for the Firehose stream.

Answer

Have the Lambda function write the transformed data directly to the S3 bucket.

Question 3

A company uses AWS Glue to perform ETL on data stored in Amazon S3. The Glue job reads CSV files, converts them to Parquet, and partitions by date. The job runs daily and processes about 500 GB of data. The team wants to optimize costs and performance. Which three actions should the team take? (Select THREE.)

Accepted Answer

Use column pruning to read only necessary columns in the Glue script.. Option B is correct because column pruning in AWS Glue scripts reduces the amount of data read from Amazon S3 by specifying only the columns needed for the ETL transformation. This minimizes I/O and network overhead, directly lowering costs and improving job performance, especially when processing large CSV files.

Answer

Increase the Spark shuffle partitions to 500.

Answer

Write the output as JSON instead of Parquet to avoid compression overhead.

Question 4

A company uses AWS Glue to process streaming data from Amazon Kinesis Data Streams. The job reads JSON records and writes Parquet to Amazon S3. Recently, the job started failing with 'Out of Memory' errors. Which change is MOST likely to resolve the issue?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. The 'Out of Memory' error in AWS Glue indicates that the job's allocated resources are insufficient for the data volume or processing complexity. Increasing the number of DPUs (Data Processing Units) directly increases the available memory and compute capacity, which is the most straightforward fix for OOM errors in Glue streaming jobs. Option C is correct because it addresses the root cause—resource exhaustion—by scaling the job horizontally.

Answer

Enable compression on the Kinesis stream.

Answer

Change the output format from Parquet to ORC.

Answer

Reduce the streaming batch size in the Glue job configuration.

Question 5

A data engineer is designing a data ingestion pipeline for IoT sensor data. The data arrives as JSON via AWS IoT Core, and must be stored in Amazon S3 in partitioned Parquet format. The pipeline must handle late-arriving data (up to 1 hour) and ensure exactly-once processing. Which combination of services should the engineer use?

Accepted Answer

Amazon Kinesis Data Firehose with data transformation via AWS Lambda, delivering to Amazon S3.. Amazon Kinesis Data Firehose is the correct choice because it can directly ingest streaming data from AWS IoT Core, use a built-in AWS Lambda function to transform JSON to Parquet, and deliver the data to Amazon S3 with automatic partitioning. It also supports buffering and retry logic to handle late-arriving data (up to 1 hour) and provides exactly-once delivery to S3 when configured with the appropriate error handling and idempotent transformations.

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation and Amazon S3.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda for transformation and Amazon S3.

Answer

AWS Glue streaming jobs consuming from Amazon Kinesis Data Streams and writing to Amazon S3.

Question 6

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3. The data volume is about 500 GB initially, with daily incremental updates of 10 GB. The pipeline must minimize operational overhead. Which AWS service should be used for the initial and incremental loads?

Accepted Answer

AWS Database Migration Service (DMS) with change data capture (CDC) to Amazon S3.. AWS DMS with CDC is the correct choice because it supports continuous replication from Oracle to Amazon S3 with minimal overhead. It handles both the initial 500 GB full load and ongoing 10 GB daily increments via change data capture, without requiring custom code or complex pipeline management.

Answer

AWS Glue with a JDBC connection and incremental crawl.

Answer

Amazon Kinesis Data Firehose with a custom producer.

Answer

AWS Data Pipeline with a SQL activity and HiveCopyActivity.

Question 7

A company has a Glue ETL job that reads from an Amazon RDS for MySQL table and writes to Amazon S3. The job runs hourly and processes new records based on a 'last_modified' timestamp column. Recently, the job started missing some records because the timestamp in MySQL is stored with microsecond precision but Glue's job bookmark only tracks second precision. Which solution addresses this issue?

Accepted Answer

Use a job parameter to store the last processed timestamp with millisecond precision and query records greater than that value.. Option A is correct because AWS Glue job bookmarks track timestamps with only second precision, so records with microsecond differences within the same second are missed. By using a custom job parameter to store the last processed timestamp with millisecond precision and querying records greater than that value, you bypass Glue's bookmark limitation and capture all new or modified records.

Answer

Increase the job frequency to every 30 minutes.

Answer

Run a full refresh of the table each time instead of incremental.

Answer

Modify the MySQL table to use a DATE data type instead of TIMESTAMP.

Question 8

A data engineer is ingesting CSV files from an Amazon S3 bucket into a Glue Data Catalog table. The files have headers, but some files have extra columns not present in the first file. The engineer wants the Glue crawler to automatically detect the schema. Which crawler configuration option should be used?

Accepted Answer

Configure the crawler to 'Create a single schema for each S3 path' and enable 'Merge tables'.. Option B is correct because when CSV files have varying schemas (extra columns), the Glue crawler must be configured to 'Create a single schema for each S3 path' with 'Merge tables' enabled. This configuration instructs the crawler to union the schemas from all files in the S3 path, adding new columns as they appear, rather than creating separate tables for each schema variation.

Answer

Configure the crawler to 'Inherit schema from table' and set the table name.

Answer

Configure the crawler to 'Create a single schema for each S3 path' without enabling 'Merge tables'.

Answer

Configure the crawler to 'Create a single schema for each S3 path' and set 'Each file as a separate table'.

Question 9

A company is building a data lake on Amazon S3. Data arrives from multiple sources in JSON, CSV, and Avro formats. The data must be transformed to Parquet and partitioned by date and source. Which TWO services can perform this transformation with minimal custom code? (Choose TWO.)

Accepted Answer

Amazon EMR with Spark. Amazon EMR with Spark is correct because Spark natively supports reading JSON, CSV, and Avro formats and writing Parquet with built-in partitioning by date and source. You can achieve this with a concise PySpark or Scala script that loads the data, applies partitioning logic, and writes to S3, requiring minimal custom code beyond the transformation logic.

Answer

AWS Lake Formation

Answer

Amazon Athena CTAS queries

Answer

Amazon Kinesis Data Firehose

Question 10

A data engineer is troubleshooting an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but 5% of records are missing after the load. The engineer suspects data consistency issues. Which THREE actions could help diagnose and resolve the problem? (Choose THREE.)

Accepted Answer

Use the Redshift COPY command with a manifest file to load data.. Option A is correct because using the Redshift COPY command with a manifest file ensures that only the exact files listed in the manifest are loaded, eliminating the risk of partial or duplicate reads from S3. This is a common pattern to guarantee data consistency when the Glue job may not reliably track which files have been processed, especially in scenarios with concurrent writes or retries.

Answer

Increase the number of DPUs for the Glue job.

Answer

Use a staging table in Redshift with a transaction to commit.

Question 11

A company uses AWS Glue to process CSV files from an S3 bucket. The job fails intermittently with a 'SchemaDetectionError' for files that have inconsistent column counts. What is the most efficient way to handle this?

Accepted Answer

Use the 'mergeSchema' option when reading the DynamicFrame.. Option A is correct because the 'mergeSchema' option in AWS Glue's DynamicFrame reader automatically reconciles schema differences across files, including inconsistent column counts. When enabled, Glue merges all schemas encountered during the read, adding nulls for missing columns in files with fewer columns, preventing the 'SchemaDetectionError' without manual intervention.

Answer

Convert all CSV files to Parquet format using a separate preprocessing job.

Answer

Define a fixed schema in the Glue job using 'apply_mapping' to map columns.

Answer

Set the job to 'ignore' schema mismatches in the job parameters.

Question 12

A data pipeline uses Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data volume spikes occasionally, causing the Firehose buffer to fill up and leading to increased delivery latency. The latency must remain under 60 seconds. What should be done to minimize latency?

Accepted Answer

Reduce the buffer interval to 60 seconds.. Reducing the buffer interval to 60 seconds ensures that Firehose delivers data to S3 at most every 60 seconds, directly capping latency even if the buffer size is not full. This aligns with the requirement to keep latency under 60 seconds, as Firehose delivers data when either the buffer interval or buffer size threshold is met first.

Answer

Enable GZIP compression on the Firehose delivery stream.

Answer

Increase the buffer size to 128 MB to accommodate larger batches.

Answer

Switch to Kinesis Data Streams with a Lambda consumer.

Question 13

A company runs a nightly AWS Glue ETL job that reads from a JDBC source (PostgreSQL) and writes to S3 in Parquet format. The job takes over 6 hours, but the SLA requires completion within 4 hours. The source table has 500 million rows and is updated frequently. Which approach will most reliably reduce job duration?

Accepted Answer

Partition the source table by year and use pushdown predicates in the Glue job.. Option C is correct because partitioning the source table by year and using pushdown predicates allows AWS Glue to read only the relevant partitions from PostgreSQL, drastically reducing the data scanned and transferred. This directly addresses the 500 million row volume and frequent updates by minimizing the JDBC read workload, which is the primary bottleneck in the 6-hour runtime.

Answer

Enable job bookmark and schedule the job to run more frequently.

Answer

Use multiple JDBC connections in parallel by setting 'hashexpression' and 'hashfield'.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 14

A company uses AWS Data Pipeline to copy data from DynamoDB to S3 daily. Recently, the pipeline started failing with 'ThrottlingException' errors. The DynamoDB table has on-demand capacity. Which action should be taken to resolve the issue?

Accepted Answer

Configure the pipeline to use a retry strategy with exponential backoff.. Option C is correct because ThrottlingException errors in AWS Data Pipeline when reading from DynamoDB indicate that the pipeline's read requests are exceeding the table's available throughput. Since the table uses on-demand capacity, which can handle spikes but has a per-second throughput limit, implementing exponential backoff in the pipeline's retry strategy allows it to reduce request rate upon throttling, aligning with AWS SDK best practices for handling DynamoDB throttling.

Answer

Increase the write capacity units of the DynamoDB table.

Answer

Replace Data Pipeline with AWS Glue using a DynamoDB connector.

Answer

Disable the pipeline's retry logic and increase the timeout.

Question 15

A data engineer needs to transform JSON data from an S3 bucket using AWS Glue. The JSON contains nested arrays and objects. Which Glue transform is best suited for flattening nested structures?

Accepted Answer

Relationalize. The Relationalize transform is specifically designed to flatten nested JSON structures (arrays and objects) into a set of related tables, making it ideal for this use case. It automatically handles complex nesting by creating separate DataFrames for each nested level and linking them via foreign keys, which is exactly what is needed when ingesting JSON with nested arrays and objects into a relational format.

Answer

Unnest

Answer

ResolveChoice

Answer

Map