DEA-C01 Data Ingestion and Transformation Practice Test 20 — 15 Questions

Question 1

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that sends data to an Amazon S3 bucket. The delivery stream has a buffer size of 5 MB and a buffer interval of 60 seconds. The data ingestion rate is 2 MB per second. The engineer notices that S3 objects are created every 60 seconds but each object is only about 2 MB. What should the engineer do to reduce the number of small S3 objects?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the buffer size to 10 MB.. Firehose flushes a buffer when either the buffer size (5 MB) or the buffer interval (60 seconds) is reached, whichever comes first. At 2 MB/s, the 60-second interval is hit first, producing ~2 MB objects (accounting for overhead) before the 5 MB threshold. Increasing the buffer size to 10 MB means the size threshold (10 MB) will be reached in ~5 seconds, but the interval still caps at 60 seconds — however, with a larger buffer, the stream accumulates more data per flush, yielding fewer, larger objects. The correct fix is to raise the buffer size so the size threshold drives flushes rather than the time interval.

Answer

Decrease the buffer interval to 30 seconds.

Answer

Reduce the buffer size to 2 MB.

Answer

Switch to Kinesis Data Streams and use a Lambda function to write to S3.

Question 2

A company uses AWS Database Migration Service (DMS) to migrate an on-premises PostgreSQL database to Amazon RDS for PostgreSQL. The migration is ongoing and uses change data capture (CDC). The engineer notices that the target database is falling behind the source by several hours. What is the MOST likely cause?

Accepted Answer

A large full load is being transferred, which delays the start of CDC.. The correct option is B: a large full load being transferred delays the start of CDC. In AWS DMS, CDC does not begin applying changes until the initial full load of the tables completes (unless you configure ongoing replication with a separate task or use the 'Start replication task' with CDC-only mode), so a long-running full load leaves the target lagging behind the source by hours. Option A is not a standard DMS setting that would cause this lag, option C is possible but less likely than the full-load bottleneck described, and option D would stop CDC entirely rather than merely cause a lag.

Answer

The target table has disabled parallel apply.

Answer

The replication instance is under-provisioned and needs to be larger.

Answer

Change data capture has been disabled on the source database.

Question 3

A data engineer is designing a data ingestion pipeline for real-time clickstream data from a website. The data must be stored in Amazon S3 in near-real time, and also be available for real-time analytics using Amazon Athena. The pipeline must handle occasional spikes of up to 10x the normal throughput. Which combination of services should the engineer use?

Accepted Answer

Amazon Kinesis Data Streams with AWS Lambda to write to Amazon S3, and Amazon Athena for queries.. Amazon Kinesis Data Streams can handle high throughput spikes (10x normal) with its scalable stream capacity. AWS Lambda can process records in near-real time and write them to Amazon S3. Amazon Athena can query the data directly from S3 using its schema-on-read approach. Option A is wrong because Amazon SQS is a pull-based queue; Lambda would need to poll SQS, adding latency and complexity for near-real-time ingestion. Option B is wrong because AWS DMS is designed for database migration and continuous replication, not for ingesting real-time clickstream data from a website. Option D is wrong because Amazon Kinesis Data Firehose is a simpler option for streaming data to S3, but the question requires near-real-time availability for Athena; Firehose has a minimum buffer interval of 60 seconds, which may not meet near-real-time requirements, and AWS Glue is typically used for batch ETL rather than low-latency streaming transformations.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda to write to Amazon S3, and Amazon Athena for queries.

Answer

AWS Database Migration Service (DMS) to stream data to Amazon S3, and Amazon Athena for queries.

Answer

Amazon Kinesis Data Firehose with AWS Glue for transformation, and Amazon Athena for queries.

Question 4

A company uses AWS Glue to run ETL jobs that process data from Amazon RDS to Amazon S3. The job runs successfully but the data in S3 is missing some records that exist in the source. The engineer notices that the job uses a JDBC connection and the query extracts data based on a timestamp column. What is the MOST likely cause of the missing records?

Accepted Answer

The timestamp column includes time portion and the job is using an exclusive upper bound.. When a Glue JDBC job extracts data using a timestamp column with an exclusive upper bound (e.g., WHERE ts < '2024-01-01 00:00:00'), any records whose timestamp equals the boundary are excluded. If the timestamp includes a time component and the job uses a date-only boundary or a strict less-than, records at the exact boundary are silently dropped, causing missing rows in S3.

Answer

The S3 bucket lacks write permissions.

Answer

The JDBC connection uses connection pooling, causing some records to be dropped.

Answer

The Glue job is configured to only read from one table.

Question 5

A data engineer is ingesting data from an on-premises database to Amazon S3 using AWS DataSync. The data transfer is scheduled to run daily at midnight. The engineer notices that the transfer takes longer than expected and sometimes does not complete before the next scheduled task. What should the engineer do to ensure the transfer completes within the window?

Accepted Answer

Increase the bandwidth limit in the DataSync task settings.. Increasing the bandwidth limit in the DataSync task settings allows more data to be transferred per second, reducing the overall transfer time. This directly addresses the issue of the transfer not completing within the window.

Answer

Decrease the bandwidth limit to reduce network congestion.

Answer

Increase the schedule frequency to every 12 hours.

Answer

Use S3 Transfer Acceleration instead of DataSync.

Question 6

A company uses Amazon Kinesis Data Firehose to deliver log data to Amazon S3. The data is transformed by a Lambda function that adds a timestamp field. Recently, the Firehose delivery stream has been failing with 'Lambda invocation failed' errors. The Lambda function's CloudWatch Logs show that the function is timing out. What is the MOST likely cause?

Accepted Answer

The Firehose buffer size is too large, causing the Lambda function to receive too many records.. The Firehose buffer size determines how many records are batched together before invoking the Lambda function. If the buffer size is too large, the Lambda function receives a payload with a large number of records, which can take longer to process than the configured timeout, causing timeouts. Since the Lambda function is timing out, the most likely cause is that the buffer size is too large, overwhelming the function. The other options are either incorrect (A, C) or less likely given the specific error (D).

Answer

The Lambda function lacks permission to write to CloudWatch Logs.

Answer

The Lambda function timeout is set to 1 minute, which is adequate.

Answer

The Lambda function is running out of memory.

Question 7

A company uses AWS Glue to process data from Amazon S3. The Glue job fails with a 'SchemaDetectionException'. The data engineer wants to ensure the schema is correctly inferred. Which TWO actions should the engineer take? (Choose two.)

Accepted Answer

Use the Glue Data Catalog as the source for schema definition.. Option A is correct because the AWS Glue Data Catalog provides a persistent, centralized metadata store where the schema (table definition, column names, and data types) can be explicitly defined, so the Glue job can read a known schema instead of relying on runtime inference that throws SchemaDetectionException. Option E is correct because setting the 'groupFiles' option to 'inPartition' in the Glue job's connection options groups many small files within each partition into a single read, which reduces the number of files the crawler/job must sample and makes schema inference more reliable and consistent. Option B is not appropriate because adding a default-valued column alters the data itself rather than fixing schema inference, and Glue's classifier would still need to detect the schema. Option C is wrong because increasing DPUs only adds compute capacity and does not resolve a schema detection failure. Option D is wrong because converting files to Parquet is a format change that does not by itself guarantee correct schema inference and may not be feasible for all sources.

Answer

Add a column with a default value to the data.

Answer

Increase the number of Glue DPUs to speed up processing.

Answer

Convert all input files to Parquet format.

Question 8

A data engineer is designing a data ingestion pipeline that uses AWS DMS to migrate data from an on-premises Oracle database to Amazon S3 in Parquet format. The engineer needs to ensure that data is continuously replicated with minimal latency. Which THREE steps should the engineer take? (Choose three.)

Accepted Answer

Configure a DMS task with a transformation rule to convert to Parquet.. Option B is correct because when creating the Amazon S3 target endpoint in AWS DMS, you must set the endpoint's data format to Parquet (via the S3 target endpoint settings) so DMS writes the migrated rows as Parquet files rather than CSV or JSON. Option D is correct because continuous replication with minimal latency requires enabling change data capture (CDC) on the source Oracle database (for example, by enabling supplemental logging and ARCHIVELOG mode) so DMS can read ongoing redo/archive log changes and apply them to S3. Option A is correct because a DMS task transformation rule (using the table-mapping 'rule-action' such as 'convert-to-parquet' or the target metadata settings) is how the task is instructed to emit Parquet output for the S3 target. Option C is not required here because AWS SCT is used for heterogeneous schema and code conversion, not for a same-engine Oracle-to-S3 data replication pipeline. Option E is incorrect because a full load only, without CDC, would not provide continuous replication and would miss ongoing changes, violating the minimal-latency continuous replication requirement.

Answer

Use AWS Schema Conversion Tool (SCT) to convert the schema.

Answer

Perform a full load only, without CDC.

Question 9

A data engineer is setting up a data ingestion pipeline using Amazon Kinesis Data Firehose to deliver web server logs to Amazon S3. The logs are in JSON format and the engineer wants to convert them to Parquet format. The engineer has configured a Glue table for the schema. However, when testing, the Firehose delivery stream fails with 'Error converting to Parquet'. The engineer checks the Glue table schema and notices that it includes a column 'timestamp' of type 'string' in the format 'yyyy-MM-dd HH:mm:ss'. The logs have a 'timestamp' field in the same format. What is the MOST likely cause of the failure?

Accepted Answer

The Glue table schema does not match the data schema exactly.. The failure is most likely due to a schema mismatch between the Glue table and the actual data. Firehose uses the Glue table schema to convert JSON to Parquet, and if the schema does not exactly match (e.g., column order, data types, or extra columns), the conversion fails. Option A is incorrect because Firehose does support Parquet conversion when a Glue table is provided. Option C is incorrect because S3 permissions would cause a different error (e.g., AccessDenied). Option D is incorrect because a string type for timestamp is acceptable as long as it matches; the issue is not the type but the overall schema mismatch.

Answer

Firehose does not support Parquet conversion.

Answer

The S3 bucket lacks write permissions for Firehose.

Answer

The 'timestamp' column must be of type 'timestamp' instead of 'string'.

Question 10

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3 for analytics. The data changes frequently and the engineer wants to capture both initial load and incremental changes with minimal latency. Which TWO AWS services should be used together? (Choose TWO.)

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is correct because it natively supports Oracle as a source endpoint and can perform both the initial full load and ongoing change data capture (CDC) from Oracle redo logs, replicating changes into Amazon S3 with minimal latency. Amazon Kinesis Data Streams is correct because it provides a low-latency streaming ingestion layer that can receive the change events and buffer them for downstream analytics consumers. Together, DMS handles the Oracle-to-AWS replication and CDC, while Kinesis Data Streams delivers the near-real-time incremental change stream. AWS Lambda is not correct because it is a compute service for event-driven code execution, not a data ingestion or replication service for Oracle CDC. AWS Glue is not correct because, while it can perform ETL and crawl data, it does not provide native Oracle CDC replication with minimal latency. AWS Transfer Family is not correct because it only supports SFTP, FTPS, and FTP file transfers into S3, not database change capture.

Answer

AWS Lambda

Answer

AWS Glue

Answer

AWS Transfer Family

Question 11

A company is using Amazon Kinesis Data Analytics for Apache Flink to process real-time clickstream data. The application reads from a Kinesis stream and writes aggregated results to an Amazon S3 bucket. The company notices that the application is falling behind and the checkpoint duration is increasing. Which THREE actions should the data engineer take to improve performance? (Choose THREE.)

Accepted Answer

Use multiple S3 prefixes in the output path to avoid throttling.. Option B is correct because Amazon S3 has per-prefix request rate limits (approximately 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix), so writing aggregated results across multiple S3 prefixes distributes the load and prevents S3 throttling from slowing the Flink sink. Option D is correct because increasing the checkpoint interval reduces how frequently the Flink application performs checkpoints, lowering checkpoint overhead and the associated processing pause, which directly addresses the increasing checkpoint duration. Option E is correct because increasing the parallelism of the Flink application adds more parallel subtasks and resources to consume the Kinesis stream and process records, allowing the application to catch up with the incoming clickstream data. Option A is incorrect because decreasing the number of shards reduces the stream's read throughput capacity, which would worsen the lag rather than improve performance. Option C is incorrect because simply increasing heap memory does not address the underlying throughput or checkpoint overhead problem and may even lengthen garbage collection pauses without raising parallelism or reducing checkpoint frequency.

Answer

Decrease the number of shards in the source Kinesis stream.

Answer

Increase the heap memory of the Flink application.

Question 12

A data engineering team is designing a near-real-time data ingestion pipeline for IoT sensor data. The data must be processed and stored in Amazon S3, with transformations applied before storage. The team needs to handle potential duplicates and ensure exactly-once processing semantics. Which TWO AWS services should be used together? (Choose TWO.)

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Streams (D) is correct because it provides a durable, scalable, near-real-time ingestion layer for IoT sensor data, and it supports exactly-once semantics when consumers use enhanced fan-out with checkpointing and deduplication logic based on sequence numbers and partition keys. Amazon Kinesis Data Analytics for Apache Flink (C) is correct because it can consume directly from Kinesis Data Streams, apply stateful transformations with exactly-once processing guarantees via Flink checkpoints, handle duplicates through keyed state and idempotent sinks, and write the transformed results to Amazon S3. Together, Kinesis Data Streams plus Kinesis Data Analytics for Apache Flink form the canonical near-real-time pipeline that ingests, transforms, deduplicates, and stores data in S3 with exactly-once semantics. Amazon Kinesis Data Firehose (A) is not selected because it is primarily a delivery service to S3 and does not natively provide the stateful exactly-once transformation and deduplication guarantees required here. Amazon SQS (B) is not selected because it is a queueing service without native stream processing or exactly-once transformation semantics for this pipeline. AWS DMS (E) is not selected because it is designed for database migration and replication, not IoT stream ingestion and transformation.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon Simple Queue Service (SQS)

Answer

AWS Database Migration Service (DMS)

Question 13

Refer to the exhibit. A data engineer is setting up an Amazon Kinesis Data Firehose delivery stream that writes to an S3 bucket named 'example-bucket'. The IAM role assumed by Firehose has the attached policy shown. When testing, the Firehose delivery stream fails with an access denied error. What is the most likely cause?

Accepted Answer

The IAM policy is missing the s3:AbortMultipartUpload and s3:ListBucket actions.. Amazon Kinesis Data Firehose requires a specific set of S3 permissions to deliver data reliably. When Firehose writes objects to S3, it uses multipart uploads for files larger than a certain size and needs to list the bucket to verify object existence and manage concurrent writes. The IAM policy shown in the exhibit likely includes s3:PutObject but omits s3:AbortMultipartUpload and s3:ListBucket. Without these, Firehose cannot complete multipart uploads or list bucket contents, resulting in an access denied error. The correct answer identifies these missing actions as the root cause.

Answer

The S3 bucket has server-side encryption enabled that needs additional permissions.

Answer

The IAM role does not have permission to use AWS KMS keys.

Answer

The bucket policy denies access from the Firehose service principal.

Question 14

A media company ingests large video files from partners via AWS Transfer Family (SFTP) into an S3 bucket. Each file is typically 2-5 GB. Once uploaded, an AWS Lambda function is triggered to transcode the video using Amazon Elastic Transcoder. The Lambda function reads the file from S3, submits a transcoding job to Elastic Transcoder, and writes the output back to a different S3 bucket. Recently, the Lambda function has been failing intermittently with timeouts, and the company reports that some files are not being transcoded. The CloudWatch logs show that the Lambda function is timing out after 15 minutes. The average transcoding job takes about 10 minutes to complete. The data engineer needs to fix the issue without changing the architecture drastically. What should the data engineer do?

Accepted Answer

Modify the Lambda function to submit the transcoding job asynchronously and exit, using an SNS topic to trigger a second Lambda function when the job completes.. The Lambda function is timing out because it waits synchronously for the Elastic Transcoder job to complete, which can exceed the 15-minute Lambda timeout. The correct fix is to decouple the transcoding submission from the waiting: modify the Lambda function to submit the job asynchronously and exit immediately, then use an SNS topic or CloudWatch Events to trigger a second Lambda function when the job completes. This avoids the timeout. Option A (increasing reserved concurrency) does not address the root cause; timeouts occur per invocation, not due to lack of concurrency. Option B (increasing timeout to 20 minutes) might work but is not ideal because jobs can vary and still exceed the new timeout; the better pattern is asynchronous processing. Option D (replacing Transfer Family with DMS) is incorrect because DMS is for database migration, not file transfer and transcoding.

Answer

Increase the Lambda function's reserved concurrency to allow multiple invocations in parallel.

Answer

Increase the Lambda function timeout to 20 minutes to accommodate longer transcoding jobs.

Answer

Replace AWS Transfer Family with AWS Database Migration Service to handle file transfers more efficiently.

Question 15

A startup is ingesting event data from a mobile app into an Amazon Kinesis Data Streams stream with 2 shards. Each shard can ingest up to 1 MB/s or 1000 records/s. The app sends about 800 records per second with an average record size of 1.5 KB. The data engineer notices that the stream is throttling some records, resulting in data loss. The engineer needs to ensure that all records are ingested without changing the application code. What should the data engineer do?

Accepted Answer

Increase the number of shards in the Kinesis stream to 3.. The stream is throttling because the total ingest capacity is exceeded. With 2 shards, the stream supports 2 MB/s and 2000 records/s. The app sends 800 records/s × 1.5 KB = 1.2 MB/s, which is under the 2 MB/s limit, but the per-shard limit is 1 MB/s. Since Kinesis distributes records across shards by partition key, if the partition key distribution is uneven, one shard may receive more than 1 MB/s, causing throttling. Adding a third shard increases total capacity to 3 MB/s and 3000 records/s, and more importantly, it provides additional shards to distribute the load, reducing the chance of any single shard exceeding its 1 MB/s limit. This requires no application code changes, as the Kinesis Client Library (KCL) or AWS SDK will automatically discover the new shards and redistribute traffic.

Answer

Reduce the average record size to below 1 KB by compressing data on the client side.

Answer

Switch from Kinesis Data Streams to Kinesis Data Firehose for ingestion.

Answer

Enable enhanced fan-out on the stream to provide dedicated throughput to each consumer.