DEA-C01 Data Ingestion and Transformation Practice Test 22 — 15 Questions

Question 1

A data engineer is designing a data ingestion pipeline using Amazon Kinesis Data Streams. The stream receives records from thousands of producers. The engineer needs to ensure that the data is durably stored and can be processed by multiple consumers. Which two actions should the engineer take to meet these requirements? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Kinesis Client Library (KCL) to build consumers that can process the stream in parallel.. On-demand capacity mode automatically scales the stream to handle varying throughput from many producers, ensuring the stream can ingest data durably. The Kinesis Client Library enables building multiple consumers that can process the stream in parallel, with each shard processed by a single consumer instance, providing fault tolerance and load balancing. Together, these actions meet the requirements.

Answer

Attach an IAM policy to the stream that allows all principals to read and write.

Answer

Enable server-side encryption on the stream to protect data at rest.

Answer

Increase the retention period of the stream to 365 days to ensure data durability.

Question 2

A data engineer is designing an ingestion pipeline for a retail company that receives point-of-sale transaction events from thousands of stores. The events must be captured in near real time, retained for 7 days so multiple consumer applications can replay them independently, and delivered to Amazon S3 for long-term analytics. Which combination of AWS services should the engineer use to meet these requirements with the LEAST operational overhead?

Accepted Answer

Amazon Kinesis Data Streams with an AWS Lambda consumer that writes batches to Amazon S3. Kinesis Data Streams is the right core service because it provides durable retention for up to 7 days by default and supports multiple independent consumers that can each read and replay the same records. Pairing it with a Lambda consumer to land data in S3 keeps the pipeline serverless and low-overhead while meeting the near-real-time and replay requirements.

Answer

Amazon Kinesis Data Firehose with a delivery stream targeting Amazon S3 and a 60-second buffer interval

Answer

Amazon Managed Streaming for Apache Kafka (MSK) with a custom Kafka Connect S3 sink connector

Answer

Amazon SQS FIFO queues with an AWS Lambda consumer that writes each message to Amazon S3

Question 3

A data engineer needs to run a daily AWS Glue extract, transform, and load (ETL) job that reads from an Amazon DynamoDB table and writes to Amazon S3 in Parquet format. The job must use the maximum read throughput available without affecting production traffic. Which read configuration should the engineer use?

Accepted Answer

Configure the AWS Glue job to use DynamoDB export to Amazon S3 and read the exported data.. DynamoDB export to Amazon S3 provides a serverless, high-throughput way to export full table data without consuming the table's read capacity. This isolates the ETL read workload from production traffic and allows AWS Glue to read the exported Parquet or DynamoDB JSON files directly from S3. It is the recommended approach for large-scale batch extracts from DynamoDB.

Answer

Enable DynamoDB Streams on the table and have the Glue job consume the stream.

Answer

Create an Amazon DynamoDB Accelerator (DAX) cluster and point the Glue job to the DAX endpoint.

Answer

Set the DynamoDB read capacity units (RCUs) to a high value and read directly from the table.

Question 4

A data engineer needs to create an AWS Glue job that transforms data from JSON to Parquet. The engineer wants to ensure the job can handle schema changes in the source data without failing. Which feature of AWS Glue should the engineer use?

Accepted Answer

DynamicFrame, which can handle schema inconsistencies and missing fields.. DynamicFrame is specifically built to handle schema variability and inconsistencies, allowing the ETL job to process data even when the schema changes. It can represent missing or unexpected fields without causing job failures, providing resilience against source data evolution.

Answer

DynamicFrame with the 'UpdateSchema' option enabled.

Answer

AWS Glue crawler with a custom classifier to detect schema changes.

Answer

Spark DataFrame with the 'mergeSchema' option set to true.

Question 5

A data engineer is orchestrating a daily ETL workflow using AWS Glue. The workflow consists of multiple dependent jobs: JobA extracts data, JobB transforms it, and JobC loads it. The engineer needs to ensure that if JobA fails, JobB and JobC do not run, and the entire workflow is retried up to two times. Which AWS Glue feature should the engineer use to implement this?

Accepted Answer

AWS Glue workflows with retry policies on the workflow. AWS Glue workflows natively support defining dependencies between jobs and crawlers. A workflow can have a retry policy that automatically retries the entire workflow if any node fails, up to a specified number of times. This ensures that if JobA fails, JobB and JobC are not executed, and the workflow retries as a whole, satisfying the requirement without external services.

Answer

AWS Step Functions state machine with AWS Glue job tasks

Answer

AWS Glue job bookmarks with retry configuration

Answer

AWS Glue triggers with conditional logic and retry settings

Question 6

A data engineer is building a nightly AWS Glue ETL job that reads from an Amazon DynamoDB table and writes to Amazon S3. The table has a simple primary key (partition key only) and is approximately 200 GB. The engineer wants to maximize read throughput while minimizing the job's runtime and avoiding impact on the production application that also uses the table. Which configuration should the engineer use for the AWS Glue DynamoDB source?

Accepted Answer

Set the connection option dynamodb.splits to a value greater than 1 so Glue performs a parallel scan across multiple segments of the table.. The AWS Glue DynamoDB connector provides the dynamodb.splits connection option to divide a table into multiple segments read concurrently. For a large table, parallel scanning is the primary way to shorten job runtime. Increasing capacity alone does not parallelize the read, and DynamoDB Streams are for change capture rather than bulk extraction, so they cannot supply the full historical dataset.

Answer

Increase the DynamoDB table's provisioned read capacity units to the maximum allowed and then run the Glue job with default settings.

Answer

Configure the Glue job to use DynamoDB parallel scans by setting the dynamodb.scan.segments and dynamodb.scan.segment properties in the connection options.

Answer

Enable DynamoDB Streams on the table and have the Glue job consume the stream with a starting position of TRIM_HORIZON.

Question 7

A data engineer is configuring an AWS Glue ETL job that reads a 2 TB Parquet dataset from Amazon S3 and writes an aggregated result to another S3 prefix. The job currently runs for 90 minutes, and the engineer wants to reduce the runtime without changing the transformation logic. The source data is partitioned by year/month/day. Which change will MOST effectively reduce the job's runtime?

Accepted Answer

Group the many small files in each partition into fewer, larger files before the job runs.. The runtime of a Glue job over partitioned S3 data is heavily influenced by file layout. Thousands of tiny objects per partition force many small read tasks and inflate overhead, so consolidating them into fewer larger files lets the same compute finish faster. Changing DPU count or adding bookmarks does not address per-file overhead, and filtering partitions would alter the required aggregate result.

Answer

Increase the number of AWS Glue DPUs allocated to the job.

Answer

Use pushdown predicates to filter the S3 partitions read by the job.

Answer

Enable job bookmarks to skip previously processed data.

Question 8

A data engineer is using AWS Glue Studio to build a job that reads JSON files from Amazon S3, applies a series of transforms, and writes Parquet to another S3 bucket. The job must handle schema evolution in the source JSON, where new fields are sometimes added and existing fields occasionally change type from integer to string. The engineer wants the job to infer the schema dynamically and continue processing without failing on type mismatches. Which approach should the engineer take?

Accepted Answer

Enable the AWS Glue job's schema inference by setting the connection option inferSchema to true and use the DynamicFrame's resolveChoice for type conflicts.. Schema evolution in AWS Glue is addressed by allowing the reader to infer the schema at runtime and by using resolveChoice to reconcile conflicting types. Inference picks up newly added fields, while resolveChoice prevents failures when a field's type changes. Static catalog schemas and crawler updates are not sufficient for continuous runtime evolution, and no single enable-schema-evolution flag exists.

Answer

Use the AWS Glue Data Catalog with a crawler configured to update the table schema, and set the job's --job-language to python with the glueetl job type.

Answer

Configure the job to use the AWS Glue DynamicFrame with the resolveChoice transform and set the transformation context to cast ambiguous types to string.

Answer

Use the AWS Glue Data Catalog table with a manually defined schema and set the job's --enable-schema-evolution argument to true.

Question 9

A data engineer needs to load a 50 GB uncompressed CSV dataset from Amazon S3 into an Amazon Redshift cluster once per day. The engineer wants the fastest load and the lowest possible storage footprint in Redshift. Which approach should the engineer use?

Accepted Answer

Use the Redshift COPY command with the FORMAT AS PARQUET option to load the Parquet files directly. The Redshift COPY command is the fastest bulk-load path, and loading Parquet directly with FORMAT AS PARQUET avoids row-by-row parsing while letting Redshift store columnar, compressed data. This meets both the fastest-load and lowest-storage-footprint requirements better than loading CSV variants or using JDBC.

Answer

Use an AWS Glue job to read the CSV and write to Redshift via JDBC in small batches

Answer

Use the Redshift COPY command with the CSV option and PARALLEL ON to split the load across slices

Answer

Use the Redshift COPY command with GZIP compression and a manifest file that lists the S3 objects

Question 10

A data engineer is building an AWS Glue ETL job that reads data from an Amazon DynamoDB table and writes it to Amazon S3 in Parquet format. The DynamoDB table has a large number of items and the engineer wants to avoid consuming all the read capacity during the job. The job must also handle items that are updated during the extraction. Which approach should the engineer use?

Accepted Answer

Use AWS Glue with a DynamoDB connection and enable the 'dynamodb.throughput.read.percent' parameter set to a low value.. AWS Glue's DynamoDB connection allows specifying the read throughput percentage via the 'dynamodb.throughput.read.percent' parameter. This controls how much of the table's read capacity the job can consume, preventing throttling of other applications. Glue also handles schema inference and can write Parquet to S3. This approach is serverless, integrated, and directly addresses the need to avoid consuming all read capacity while extracting data.

Answer

Use Amazon EMR with a custom MapReduce job that reads from DynamoDB using the DynamoDB connector and writes to S3.

Answer

Use Amazon DynamoDB Streams to capture changes and process them with AWS Lambda, writing to S3.

Answer

Use AWS Database Migration Service (AWS DMS) with a DynamoDB source and S3 target, enabling change data capture (CDC).

Question 11

A data engineer is using AWS Glue to process a large dataset stored in Amazon S3 in Parquet format. The Glue job performs a join between two DynamicFrames: one large (10 TB) and one small (10 MB). The engineer notices the job is slow and often fails with out-of-memory errors. Which technique should the engineer use to optimize the join performance?

Accepted Answer

Broadcast the small DynamicFrame to all worker nodes using the `broadcast` method before joining.. When joining a large dataset with a very small one, broadcasting the small dataset to all worker nodes enables a map-side join, avoiding a costly shuffle of the large dataset. This reduces network I/O and memory pressure, preventing out-of-memory errors. AWS Glue's `Join` transform can automatically broadcast if configured, but explicitly broadcasting the small frame is a reliable optimization.

Answer

Convert both DynamicFrames to DataFrames and use the `join` method with a shuffle hash join.

Answer

Use the `Join` transform with the `transformation_ctx` parameter set to a unique string.

Answer

Increase the number of DPUs for the Glue job to 100.

Question 12

A data engineer is building a streaming ingestion pipeline that reads records from an Amazon Kinesis Data Stream and writes them to Amazon S3 in Parquet format. The records must be delivered within 60 seconds and the schema must be discovered automatically from the stream. Which combination of AWS services should the engineer use to meet these requirements with the LEAST operational overhead?

Accepted Answer

Kinesis Data Firehose with AWS Glue Data Catalog schema inference and Parquet conversion enabled. Kinesis Data Firehose is the managed service designed for this use case: it buffers incoming records for a configurable interval (as low as 60 seconds), converts JSON to Parquet using the AWS Glue Data Catalog, and can infer schema automatically. This removes the need to manage consumers, checkpointing, or Flink/Glue job code, directly satisfying the low-latency and low-overhead requirements.

Answer

AWS Lambda consumer reading from Kinesis with the Kinesis Client Library writing Parquet to S3

Answer

Amazon Managed Service for Apache Flink application with a S3 sink and schema inference

Answer

AWS Glue streaming ETL job reading from Kinesis and writing Parquet to S3

Question 13

A data engineer has an AWS Glue ETL job that reads JSON records from Amazon S3 and writes them to Amazon Redshift. The job intermittently fails with a 'Spectrum Scan Error' when the source partition contains a malformed record. The engineer wants the job to skip malformed records and log them for later analysis without failing the entire run. What should the engineer do?

Accepted Answer

Create the DynamicFrame with the 'errorBadRecords' option set to true so bad rows are routed to an error collection.. Glue DynamicFrame creation supports the errorBadRecords option, which routes unparseable records into a dedicated errors collection rather than failing the job. Writing that collection to S3 preserves the bad rows for later inspection. Bookmarking, metrics, and ResolveChoice address different problems: reprocessing, observability, and type conflicts respectively, and none of them handle syntactically invalid input.

Answer

Enable job bookmarks on the Glue job and set the transformation_ctx parameter to the source path.

Answer

Set the Glue job's '--enable-metrics' flag and increase the number of DPUs allocated to the job.

Answer

Use the Glue DynamicFrame ResolveChoice transformation with the 'make_struct' option on the source frame.

Question 14

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3 on a daily basis. The data volume is large (several terabytes) and the engineer wants to minimize the impact on the source database. The data must be stored in Parquet format for querying with Amazon Athena. Which AWS service should the engineer use to perform the ingestion?

Accepted Answer

AWS Glue with a JDBC connection to the Oracle database, reading the data and writing to S3 in Parquet format.. AWS Glue can connect to an on-premises Oracle database using JDBC, read data in parallel, and write it to S3 in Parquet format. It is serverless and designed for batch ETL, making it the best fit for daily ingestion of large data volumes with minimal impact on the source. Other services are either for streaming, file copying, or continuous replication.

Answer

AWS DataSync to copy Oracle database files directly to Amazon S3.

Answer

Amazon Kinesis Data Firehose with a custom producer application reading from Oracle.

Answer

AWS Database Migration Service (AWS DMS) with ongoing replication to Amazon S3.

Question 15

A data engineer needs to run a transformation on streaming data from Amazon Kinesis Data Streams and write the results to Amazon S3 in near real-time. The transformation is a simple stateless mapping that can be expressed in SQL. The engineer wants to minimize operational overhead and avoid managing servers. Which service should the engineer use?

Accepted Answer

Amazon Managed Service for Apache Flink with a SQL application using Kinesis Data Streams as source and Amazon S3 as sink.. Amazon Managed Service for Apache Flink provides a serverless environment for running SQL applications that process streaming data. It can read from Kinesis Data Streams and write to S3, satisfying the near real-time and SQL transformation requirements. The other options either require custom code, server management, or are better suited for batch or buffered delivery rather than a simple SQL stream mapping.

Answer

Amazon Kinesis Client Library (KCL) application running on Amazon EC2 instances that reads from Kinesis and writes to S3.

Answer

Amazon Kinesis Data Firehose with an AWS Lambda function for record transformation.

Answer

AWS Glue Streaming ETL job with a Kinesis Data Streams source and an S3 sink, written in PySpark.