DEA-C01 Data Ingestion and Transformation Practice Test 15 — 15 Questions

Question 1

A company uses Kinesis Data Streams to ingest clickstream data. They notice that the data processing latency increases as the number of shards grows. What is the most likely cause and solution?

Accepted Answer

Reduce the number of shards or increase the number of consumers.. As the number of shards increases, the total throughput of the stream increases, but each shard has a fixed limit of 5 read transactions per second. If the number of consumers remains constant, each consumer must poll more shards sequentially, increasing the per-record processing latency. Reducing the number of shards or adding more consumers distributes the polling load, reducing the time each consumer spends waiting for shard-level throttling.

Answer

Increase the Kinesis Producer Library (KPL) batch size.

Answer

Use enhanced fan-out to allow multiple consumers per shard.

Answer

Increase the number of shards to handle more data.

Question 2

A data engineer needs to ingest JSON files from an S3 bucket into a DynamoDB table. The files are updated hourly and contain new records. Which AWS service should be used to trigger a Lambda function for each new object?

Accepted Answer

S3 Event Notifications. S3 Event Notifications are the correct choice because they are designed to trigger AWS Lambda functions directly in response to object creation events (e.g., `s3:ObjectCreated:*`) in an S3 bucket. This allows the data engineer to automatically invoke a Lambda function for each new JSON file as it is uploaded, enabling ingestion into DynamoDB without polling or additional infrastructure.

Answer

Kinesis Data Firehose

Answer

Amazon EventBridge

Answer

Amazon SQS

Question 3

A company uses AWS Glue ETL jobs to transform data in S3. The job runs successfully but takes longer than expected. The data is in Parquet format and partitioned by date. Which change would most improve performance without increasing cost?

Accepted Answer

Enable pushdown predicates to filter partitions early.. Pushdown predicates allow AWS Glue to filter data at the storage layer (e.g., S3 partition pruning) before reading it into memory. Since the data is partitioned by date, enabling pushdown predicates reduces the amount of data scanned, which directly decreases job runtime without requiring additional DPUs or changing the data format.

Answer

Repartition the data by a different column.

Answer

Convert Parquet to CSV for faster serialization.

Answer

Increase the number of DPUs for the job.

Question 4

A data engineer is designing a streaming pipeline that ingests IoT sensor data from 10,000 devices. Each device sends a 1 KB message every second. The data must be processed in near real-time and stored in S3 for analytics. Which combination of services provides the most cost-effective solution?

Accepted Answer

Amazon Kinesis Data Streams with Kinesis Data Firehose delivery to S3.. B is correct because Kinesis Data Streams ingests high-throughput IoT data (10,000 messages/sec at 1 KB each) with low latency, and Kinesis Data Firehose automatically batches and compresses data before delivering it to S3, eliminating the need for custom code or manual scaling. This combination provides the most cost-effective near-real-time solution by leveraging Firehose's built-in buffering and compression to minimize S3 storage costs and reduce the number of PUT requests.

Answer

AWS Data Pipeline with periodic S3 copy.

Answer

Amazon MSK (Managed Streaming for Kafka) with Kafka Connect S3 sink.

Answer

Amazon SQS FIFO queue with Lambda consumers writing to S3.

Question 5

A company runs a nightly ETL job using AWS Glue. The job reads data from a JDBC connection to an on-premises MySQL database. The job fails with an error indicating that the connection pool is exhausted. What is the most likely cause and solution?

Accepted Answer

The Glue job is using too many executors, exhausting the database connections. Reduce the number of DPUs or increase the database max connections.. AWS Glue jobs distribute work across multiple executors, each of which opens its own JDBC connection to the source database. When the number of executors (controlled by DPUs) exceeds the database's configured maximum connections, the database connection pool is exhausted, causing the error. Reducing the number of DPUs or increasing the database's max_connections setting resolves the issue.

Answer

The database is not reachable due to network issues. Check VPC and security groups.

Answer

The Glue job is hitting the AWS Glue connection pool limit. Increase the Glue connection pool size.

Answer

The database credentials are expired. Rotate the password in AWS Secrets Manager.

Question 6

A data engineer needs to transform CSV files to Parquet format using AWS Glue. The source data contains sensitive columns that must be masked. Which Glue feature should be used?

Accepted Answer

AWS Glue DataBrew. AWS Glue DataBrew is a visual data preparation tool that includes built-in transformations for masking sensitive data, such as hashing, tokenizing, or redacting columns. This makes it the correct choice for transforming CSV files to Parquet while applying column-level masking without writing custom code.

Answer

AWS Glue Studio

Answer

AWS Glue Crawler

Answer

AWS Glue Schema Registry

Question 7

A data pipeline ingests streaming data from Kinesis Data Streams into S3 via Kinesis Data Firehose. Occasionally, small files are written to S3, increasing downstream processing costs. What is the most efficient way to reduce the number of small files?

Accepted Answer

Increase the Firehose buffering interval to 300 seconds and buffering size to 64 MB.. Kinesis Data Firehose allows you to configure buffering hints (size and interval) to control when data is delivered to S3. By increasing the buffering interval to 300 seconds and the buffering size to 64 MB, Firehose accumulates more records before writing, which reduces the number of small files. This is the most efficient approach as it requires no additional infrastructure or post-processing.

Answer

Use a Lambda function to aggregate records before sending to Firehose.

Answer

Use the Kinesis Client Library (KCL) to write larger batches to S3 directly.

Answer

Run a daily AWS Glue job to concatenate small files.

Question 8

A company uses AWS Database Migration Service (DMS) to continuously replicate data from an Oracle RDS instance to S3. The data is used for analytics. The replication lags behind the source by several hours. Which change would most likely reduce the lag?

Accepted Answer

Use a larger DMS replication instance (e.g., dms.c5.large instead of dms.t3.medium).. The replication lag is most likely caused by the DMS replication instance being undersized for the volume of change data capture (CDC) events. Upgrading from a burstable t3.medium to a compute-optimized c5.large instance provides more consistent CPU performance and higher network throughput, enabling faster processing of Oracle redo logs and reducing the lag between source and target.

Answer

Change the target endpoint from S3 to Kinesis Data Firehose.

Answer

Increase the source RDS instance storage to improve I/O.

Answer

Change the target data format from CSV to Parquet.

Question 9

A data engineer needs to ingest data from an external FTP server into S3 on a schedule. The FTP server is only accessible via VPN. Which AWS service is best suited for this task?

Accepted Answer

AWS Transfer Family. AWS Transfer Family supports FTP, FTPS, and SFTP protocols and can be integrated with a VPC using Elastic IPs or a VPC endpoint, enabling secure access to an FTP server reachable only via VPN. It automates the transfer of files from the external FTP server to Amazon S3 on a schedule without requiring custom code or infrastructure management.

Answer

AWS Snowcone

Answer

AWS Glue with a Python shell

Answer

AWS DataSync

Question 10

A data engineer is designing a near-real-time streaming pipeline to ingest clickstream data from a web application. The data must be enriched with user metadata from a DynamoDB table before being stored in S3. Which combination of AWS services should the engineer use? (Choose TWO.)

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Streams (C) provides the low-latency, durable ingestion layer for the clickstream data, while Amazon Kinesis Data Analytics for Apache Flink (B) allows you to run a Flink application that can perform stream-to-stream joins with the DynamoDB user metadata in near-real time. The enriched output can then be written to S3 via a Kinesis Data Firehose delivery stream or directly from the Flink application.

Answer

Amazon Kinesis Data Firehose

Answer

AWS Lambda with DynamoDB Accelerator (DAX)

Answer

AWS Glue Streaming ETL

Question 11

A company uses a Kinesis Data Firehose delivery stream to load data into an S3 bucket. The data is in JSON format and must be converted to Parquet before landing in S3. Which steps are required to achieve this? (Choose THREE.)

Accepted Answer

Configure the Firehose delivery stream to enable data format conversion to Parquet.. Kinesis Data Firehose natively supports converting incoming data from JSON to Parquet format. This conversion is enabled directly in the delivery stream configuration, eliminating the need for separate processing steps.

Answer

Store the schema in Amazon DynamoDB.

Answer

Use Kinesis Data Analytics to convert the data.

Question 12

A data engineer needs to ingest data from a SaaS application that sends webhooks in JSON format. The data must be stored in S3 for batch analysis. Which AWS services can receive the webhooks and store the data in S3 with minimal custom code? (Choose TWO.)

Accepted Answer

Amazon API Gateway with S3 integration. Amazon API Gateway with S3 integration (Option D) allows you to create a REST API that directly writes incoming webhook payloads to an S3 bucket without any custom code, using a proxy integration with an AWS service. This minimizes custom code because the integration handles the mapping and storage automatically.

Answer

AWS Lambda with S3 SDK

Answer

AWS Glue with a Python shell job

Answer

Amazon Kinesis Data Streams

Question 13

Refer to the exhibit. A data engineer is configuring an IAM policy for a Lambda function that writes transformed data to S3. The function writes to both 'example-bucket/data/' and 'example-bucket/public/'. The policy is intended to enforce server-side encryption with SSE-S3 for all objects written to the 'public/' prefix, while allowing all operations on other prefixes. However, the Lambda function is failing with an AccessDenied error when writing to 'example-bucket/public/'. What is the most likely cause?

Accepted Answer

The Lambda function is not setting the 'x-amz-server-side-encryption' header to 'AES256' when writing to 'public/'.. The policy enforces SSE-S3 encryption for objects written to the 'public/' prefix. When a Lambda function writes to S3 without setting the 'x-amz-server-side-encryption' header to 'AES256', the request fails with an AccessDenied error if the bucket policy requires SSE-S3. The policy explicitly denies PutObject unless the encryption header is present, so the function must include this header to succeed.

Answer

The policy denies DeleteObject on 'public/'.

Answer

The policy denies PutObject on 'public/' unconditionally.

Answer

The policy does not allow GetObject for 'public/'.

Question 14

Refer to the exhibit. A data engineer runs the above CLI command to find files smaller than 1000 bytes in a bucket. The command returns an empty array, but the engineer knows there are small files. What is the issue?

Accepted Answer

The Size is compared as a string, not an integer; remove quotes around '1000'.. In the AWS CLI `list-objects-v2` command with `--query`, the `Size` field is a numeric value, but the query string `Size < '1000'` compares it as a string. This causes a lexicographic comparison, so files with sizes like '900' would be correctly matched, but any size with more digits (e.g., '1000' itself or '999') may fail due to string ordering. Removing the quotes around `1000` treats it as an integer, enabling proper numeric comparison.

Answer

The prefix is incorrect; it should be 'logs/2023/01/01/'.

Answer

The bucket policy does not allow listing objects.

Answer

The query syntax is invalid; use a filter instead.

Question 15

Refer to the exhibit. A data engineer deploys this CloudFormation template to create an AWS Glue job. The job fails on the first run with an error: 'AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/GlueServiceRole/... is not authorized to perform: s3:GetObject on resource: s3://my-bucket/scripts/etl.py'. What is the most likely cause?

Accepted Answer

The IAM role associated with the Glue job does not have an S3 GetObject permission for the script location.. The error message indicates that the IAM role 'GlueServiceRole' assumed by the AWS Glue job does not have the s3:GetObject permission for the script object at s3://my-bucket/scripts/etl.py. AWS Glue requires the execution role to have read access to the script location specified in the 'ScriptLocation' parameter. Without this permission, the job fails immediately on startup because it cannot download and execute the ETL script.

Answer

The ExecutionProperty MaxConcurrentRuns is set to 1, preventing the job from running.

Answer

The MaxRetries is set to 0, so the job does not retry on failure.

Answer

The script location is incorrectly specified; it should be an S3 URI with bucket and key.