DEA-C01 Data Ingestion and Transformation Practice Test 40 — 15 Questions

Question 1

A company is using Amazon Kinesis Data Analytics for Apache Flink to process real-time clickstream data. The application reads from a Kinesis stream and writes aggregated results to an Amazon S3 bucket. The company notices that the application is falling behind and the checkpoint duration is increasing. Which THREE actions should the data engineer take to improve performance? (Choose THREE.)

Accepted Answer

Use multiple S3 prefixes in the output path to avoid throttling.. Options A, C, and E are correct. Increasing parallelism allows more parallel processing. Using S3 with multiple prefixes reduces S3 write throttling. Increasing checkpoint interval reduces overhead. Option B is wrong because heap memory increase is not directly related to checkpoint duration. Option D is wrong because reducing the number of shards would decrease throughput.

Answer

Decrease the number of shards in the source Kinesis stream.

Answer

Increase the heap memory of the Flink application.

Question 2

A data engineering team is designing a near-real-time data ingestion pipeline for IoT sensor data. The data must be processed and stored in Amazon S3, with transformations applied before storage. The team needs to handle potential duplicates and ensure exactly-once processing semantics. Which TWO AWS services should be used together? (Choose TWO.)

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Options B and C are correct. Kinesis Data Streams provides ordered data and can integrate with Kinesis Data Analytics for Flink, which supports exactly-once semantics. Option A (Firehose) provides at-least-once delivery. Option D (SQS) is for decoupling but does not provide exactly-once. Option E (DMS) is for database replication.

Answer

Amazon Kinesis Data Firehose

Answer

Amazon Simple Queue Service (SQS)

Answer

AWS Database Migration Service (DMS)

Question 3

Refer to the exhibit. A data engineer is setting up an Amazon Kinesis Data Firehose delivery stream that writes to an S3 bucket named 'example-bucket'. The IAM role assumed by Firehose has the attached policy shown. When testing, the Firehose delivery stream fails with an access denied error. What is the most likely cause?

Accepted Answer

The IAM policy is missing the s3:AbortMultipartUpload and s3:ListBucket actions.. Option A is correct because Kinesis Data Firehose requires the s3:AbortMultipartUpload and s3:ListBucket permissions to write to S3. The policy only allows s3:PutObject and s3:GetObject. Option B is wrong because SSE is not required. Option C is wrong because the bucket policy is not shown to be restrictive. Option D is wrong because encryption keys are not mentioned.

Answer

The S3 bucket has server-side encryption enabled that needs additional permissions.

Answer

The IAM role does not have permission to use AWS KMS keys.

Answer

The bucket policy denies access from the Firehose service principal.

Question 4

A media company ingests large video files from partners via AWS Transfer Family (SFTP) into an S3 bucket. Each file is typically 2-5 GB. Once uploaded, an AWS Lambda function is triggered to transcode the video using Amazon Elastic Transcoder. The Lambda function reads the file from S3, submits a transcoding job to Elastic Transcoder, and writes the output back to a different S3 bucket. Recently, the Lambda function has been failing intermittently with timeouts, and the company reports that some files are not being transcoded. The CloudWatch logs show that the Lambda function is timing out after 15 minutes. The average transcoding job takes about 10 minutes to complete. The data engineer needs to fix the issue without changing the architecture drastically. What should the data engineer do?

Accepted Answer

Modify the Lambda function to submit the transcoding job asynchronously and exit, using an SNS topic to trigger a second Lambda function when the job completes.. Option B is correct because the Lambda function should not wait for the transcoding job to complete; it should submit the job and exit. The current design synchronously waits, causing timeouts. Option A is wrong because increasing the timeout to 20 minutes may still fail if jobs take longer. Option C is wrong because the issue is not about concurrency. Option D is wrong because DMS is for database migration, not file processing.

Answer

Increase the Lambda function's reserved concurrency to allow multiple invocations in parallel.

Answer

Increase the Lambda function timeout to 20 minutes to accommodate longer transcoding jobs.

Answer

Replace AWS Transfer Family with AWS Database Migration Service to handle file transfers more efficiently.

Question 5

A financial services company is building a real-time fraud detection system. Transaction data is ingested via Amazon Kinesis Data Streams and processed by an Amazon Kinesis Data Analytics for Apache Flink application that runs sliding window aggregations. The output is written to an Amazon S3 bucket for downstream analysis. The Flink application is configured with parallelism of 4 and checkpointing every minute. The company has noticed that the application is experiencing high latency and the checkpointing is frequently failing. The CloudWatch metrics show that the Flink application's CPU utilization is near 100% and the checkpoint duration is spiking to over 5 minutes. The data engineer needs to improve performance. Which action should the data engineer take?

Accepted Answer

Increase the parallelism of the Flink application to distribute the workload across more resources.. Option C is correct because increasing parallelism allows the workload to be distributed across more resources, reducing CPU pressure and checkpoint duration. Option A is wrong because reducing checkpoint interval would increase frequency and likely worsen failures. Option B is wrong because increasing shards without increasing parallelism may not help if CPU is bottleneck. Option D is wrong because memory increase is not the primary issue.

Answer

Increase the number of shards in the source Kinesis stream to improve throughput.

Answer

Increase the heap memory of the Flink application to handle larger state.

Answer

Decrease the checkpoint interval to 30 seconds to reduce the amount of state being checkpointed.

Question 6

A startup is ingesting event data from a mobile app into an Amazon Kinesis Data Streams stream with 2 shards. Each shard can ingest up to 1 MB/s or 1000 records/s. The app sends about 800 records per second with an average record size of 1.5 KB. The data engineer notices that the stream is throttling some records, resulting in data loss. The engineer needs to ensure that all records are ingested without changing the application code. What should the data engineer do?

Accepted Answer

Increase the number of shards in the Kinesis stream to 3.. Option C is correct because the current throughput is 800 records/s * 1.5 KB = 1.2 MB/s, exceeding the 1 MB/s per shard limit. Adding a shard increases capacity. Option A is wrong because decreasing record size is not possible without code change. Option B is wrong because enhanced fan-out is for consumers, not producers. Option D is wrong because the bottleneck is ingestion, not processing.

Answer

Reduce the average record size to below 1 KB by compressing data on the client side.

Answer

Switch from Kinesis Data Streams to Kinesis Data Firehose for ingestion.

Answer

Enable enhanced fan-out on the stream to provide dedicated throughput to each consumer.

Question 7

A company uses AWS Glue to run ETL jobs that transform data from an Amazon S3 bucket (raw) to another S3 bucket (curated). The jobs run on a schedule and process data incrementally. The data engineer notices that the jobs are taking longer to complete each day, and the job metrics show that the number of DPUs (Data Processing Units) is underutilized. The engineer wants to improve job performance. What should the data engineer do?

Accepted Answer

Increase the number of DPUs allocated to the Glue job to enable more parallelism.. Option A is correct because increasing the number of DPUs can reduce job duration if the workload is parallelizable. The underutilization suggests that more DPUs could be used. Option B is wrong because decreasing DPUs would increase time. Option C is wrong because the issue is not about job bookmarks (which handle incremental processing). Option D is wrong because converting to streaming may not be appropriate for batch jobs.

Answer

Switch from batch processing to streaming using AWS Glue Streaming.

Answer

Enable job bookmarks to skip already processed data more efficiently.

Answer

Decrease the number of DPUs to reduce resource contention.

Question 8

A logistics company ingests GPS tracking data from thousands of vehicles into Amazon S3 via AWS Direct Connect. Each vehicle sends a message every 5 seconds, resulting in about 200,000 messages per second. Each message is about 200 bytes. The company uses AWS Glue to transform the data into a parquet format and load it into Amazon Redshift for real-time analytics. However, the Glue jobs are failing due to memory issues and the data is not being loaded into Redshift quickly enough. The company needs to reduce the latency of data availability in Redshift. Which action should the data engineer take?

Accepted Answer

Use Amazon Kinesis Data Firehose to ingest the data directly into S3 and then use Redshift Spectrum to query the data without loading.. Option B is correct because Amazon Kinesis Data Firehose can ingest high-throughput streaming data and deliver it to S3 in near-real-time, reducing latency. Option A is wrong because increasing Glue DPUs may not fix memory issues and doesn't reduce latency significantly. Option C is wrong because Kinesis Data Analytics adds processing overhead. Option D is wrong because Redshift Spectrum is for querying S3, not reducing ingestion latency.

Answer

Use Amazon Kinesis Data Analytics to process the data in real-time and write to Redshift directly.

Answer

Increase the size of the Redshift cluster to improve load performance.

Answer

Increase the number of DPUs and allocate more memory to the Glue job.

Question 9

A data engineer is responsible for ingesting daily CSV files from an external partner into an Amazon S3 bucket. The partner uploads files to an AWS Transfer Family (SFTP) endpoint. Once a file is uploaded, an AWS Lambda function triggers an AWS Glue ETL job to transform the data and load it into an Amazon RDS database. Recently, some files have failed to trigger the Glue job because the Lambda function timed out while waiting for the Glue job to complete. The engineer needs to ensure that all files are processed reliably without manual intervention. What should the data engineer do?

Accepted Answer

Modify the Lambda function to send a message to an Amazon SQS queue after uploading, and create a separate Lambda function that reads from the queue and triggers the Glue job asynchronously.. Option A is correct because decoupling the Lambda function from the Glue job by using SQS allows the Lambda function to submit the job and exit, while a second Lambda function monitors completion. Option B is wrong because increasing Lambda timeout still ties it to job duration. Option C is wrong because the issue is not concurrency. Option D is wrong because increasing S3 events does not address the timeout.

Answer

Increase the Lambda function timeout to 15 minutes to accommodate longer Glue jobs.

Answer

Increase the Lambda function's reserved concurrency to allow multiple invocations.

Answer

Configure S3 event notifications to trigger the Glue job directly without Lambda.

Question 10

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data from a website. The data is consumed by an Amazon Kinesis Data Analytics for Apache Flink application that performs real-time analytics. The Flink application writes its results to an Amazon S3 bucket. The company has noticed that the Flink application is experiencing high checkpoint failure rates, causing delays. The CloudWatch metrics show that the checkpoint size is large and increasing. The data engineer needs to reduce the checkpoint size. Which action should the data engineer take?

Accepted Answer

Enable incremental checkpointing in the Flink application to only write changes since the last checkpoint.. Option D is correct because enabling incremental checkpointing in Flink reduces the amount of data written per checkpoint by only writing changes since the last checkpoint. Option A is wrong because reducing parallelism may increase load per operator. Option B is wrong because decreasing checkpoint interval increases frequency, not size. Option C is wrong because state TTL does not directly reduce checkpoint size.

Answer

Decrease the checkpoint interval to reduce the amount of state accumulated.

Answer

Reduce the parallelism of the Flink application.

Answer

Increase the state time-to-live (TTL) configuration to retain state longer.

Question 11

A healthcare company is ingesting patient data from a legacy system into an Amazon S3 data lake using AWS Glue. The legacy system produces CSV files with inconsistent schemas (columns may appear or disappear in different files). The data engineer needs to create a Glue ETL job that can handle schema evolution and transform the data into a standardized parquet format. The job should also be able to process new files as they arrive. Which approach should the data engineer use?

Accepted Answer

Use AWS Glue DynamicFrames to read the CSV files and apply transformations using resolveChoice and applyMapping.. Option C is correct because Glue DynamicFrames can handle schema evolution by allowing schema-on-read and resolving schema inconsistencies. Option A is wrong because crawlers are for cataloging, not ETL. Option B is wrong because explicit schema mapping would fail with evolving schemas. Option D is wrong because Spark DataFrames require a predefined schema.

Answer

Use AWS Glue crawlers to create a schema in the Data Catalog and then use a standard Spark DataFrame for transformation.

Answer

Use a Python shell job in Glue to manually parse each file and write to parquet.

Answer

Use a Glue ETL job with a static schema defined in the script and ignore files that don't match.

Question 12

A company is using Amazon Kinesis Data Firehose to ingest log data from web servers into an Amazon S3 bucket. The data is then queried by Amazon Athena. The company has noticed that the Athena queries are slow and expensive. The data engineer wants to optimize the storage format to improve query performance and reduce costs. Which configuration change should the data engineer make to the Firehose delivery stream?

Accepted Answer

Enable data transformation in Firehose to convert JSON to Parquet format with Snappy compression.. Option B is correct because converting data to Parquet format and compressing it reduces storage space and improves query performance in Athena. Option A is wrong because storing as ORC is also good but Parquet is more common with Athena. Option C is wrong because increasing buffer size delays delivery. Option D is wrong because enabling S3 server access logs adds cost and does not help query performance.

Answer

Increase the buffer interval to 600 seconds and buffer size to 128 MB to create larger files.

Answer

Change the output format to ORC and enable GZIP compression.

Answer

Enable S3 server access logs to track query patterns.

Question 13

A data engineer is designing a data ingestion pipeline for a social media company. The pipeline ingests user posts from a REST API into Amazon S3. The API returns JSON data with an array of posts. The engineer needs to transform the data into individual JSON objects per post and store them in S3 with a partition structure of year/month/day/hour. The data should be available in S3 within 15 minutes of ingestion. The engineer decides to use AWS Lambda for transformation. Which combination of services should the engineer use to meet these requirements with minimal operational overhead?

Accepted Answer

Use Amazon CloudWatch Events to trigger an AWS Lambda function every 15 minutes. The Lambda function calls the API, transforms the data, and writes individual JSON objects to S3 with the required partition structure.. Option A is correct because using CloudWatch Events to trigger Lambda on a schedule reduces complexity and Lambda can fetch API data and write to S3. Option B is wrong because Kinesis Firehose expects streaming data, not batch API calls. Option C is wrong because Glue is heavier for simple transformation. Option D is wrong because Step Functions adds unnecessary complexity.

Answer

Use AWS Step Functions to orchestrate an API call and data transformation with Lambda, running every 15 minutes.

Answer

Use AWS Glue ETL jobs scheduled with AWS Glue triggers to run every 15 minutes.

Answer

Use Amazon Kinesis Data Firehose with a Lambda function for transformation. Configure Firehose to pull from the API every 15 minutes.

Question 14

A company is using AWS Glue to catalog data in Amazon S3. The data is in CSV format with varying schemas. The Data Engineering team wants to ensure the Glue Data Catalog is updated automatically when new partitions are added to S3. Which TWO actions should be taken? (Choose two.)

Accepted Answer

Enable partition indexing on the Glue Data Catalog.. A is correct because enabling partition indexing in the Glue Data Catalog allows partition pruning and automatic updates. C is correct because configuring a Glue crawler with a schedule will automatically discover new partitions. B is wrong because setting up an S3 event notification to trigger Lambda for manual updates is not as efficient as crawler scheduling. D is wrong because using AWS Glue ETL jobs to update the catalog is not automatic. E is wrong because Amazon Athena does not update the catalog.

Answer

Set up an S3 event notification to trigger a Lambda function that updates the Glue Data Catalog.

Answer

Run Amazon Athena queries with MSCK REPAIR TABLE to add partitions.

Answer

Use AWS Glue ETL jobs to write data and update the catalog simultaneously.

Question 15

A financial services company is ingesting real-time stock trade data into Amazon Kinesis Data Streams. The data is then processed by a Kinesis Data Analytics application for fraud detection. The company must ensure that the data is processed in the correct order for each stock symbol. Which TWO configuration steps should be taken? (Choose two.)

Accepted Answer

Use the stock symbol as the partition key when putting records into the stream.. A is correct because using a partition key of stock symbol ensures records with the same symbol go to the same shard, preserving order. D is correct because enabling the Kinesis Client Library (KCL) with sequence number ordering ensures records are processed in order within a shard. B is wrong because increasing shards does not guarantee ordering. C is wrong because using a random partition key would distribute records across shards, breaking order. E is wrong because DynamoDB Streams is for CDC, not for streaming order.

Answer

Use a random partition key to distribute the load evenly.

Answer

Increase the number of shards to reduce latency.

Answer

Use AWS Lambda with DynamoDB Streams instead of Kinesis Data Analytics.