DEA-C01 Data Ingestion and Transformation Practice Test 28 — 15 Questions

Question 1

A company uses Amazon Kinesis Data Analytics for real-time anomaly detection on clickstream data. The application uses a sliding window of 1 minute. The data engineer notices that the application is producing incorrect results because late-arriving records are not being handled properly. What should the data engineer do to ensure late records are included in the window calculations?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the watermark delay in the Kinesis Data Analytics application to allow more time for late records.. Kinesis Data Analytics uses watermarks to track event time progress and determine when to finalize window calculations. Increasing the watermark delay allows the application to wait longer for late-arriving records before closing the window, ensuring they are included in the aggregation.

Answer

Use a Kinesis Data Firehose to buffer the data and then send to Kinesis Data Analytics.

Answer

Increase the window size from 1 minute to 2 minutes.

Answer

Increase the retention period of the Kinesis stream to 7 days.

Question 2

A company needs to ingest data from a self-managed Apache Kafka cluster running on EC2 into Amazon S3. The data must be delivered in near real-time. Which AWS service is BEST suited for this task?

Accepted Answer

Use an AWS Glue streaming ETL job that reads from the Kafka cluster and writes to S3.. AWS Glue streaming ETL jobs can connect directly to an Apache Kafka cluster (including self-managed) as a source and write data to Amazon S3 in near real-time, making it a fully managed and suitable solution. Option A (Amazon MSK) is a managed Kafka service but does not directly ingest into S3; Option B (S3 Transfer Acceleration) only accelerates uploads, not ingestion; Option C (Kinesis Data Streams) adds unnecessary complexity as an intermediary.

Answer

Use Amazon MSK to replicate the Kafka cluster and then use a connector to S3.

Answer

Use Amazon S3 Transfer Acceleration to speed up the transfer from Kafka brokers to S3.

Answer

Use Amazon Kinesis Data Streams as an intermediary to buffer data before writing to S3.

Question 3

A data engineer is designing a pipeline to ingest daily CSV files from an SFTP server into Amazon S3. The files are large (up to 10 GB) and must be encrypted in transit. The pipeline should be fully managed and serverless where possible. Which TWO services should be used together to achieve this? (Choose TWO.)

Accepted Answer

AWS Glue. Options C and D are correct. AWS Transfer Family provides a fully managed SFTP service that can transfer files directly to S3 with encryption in transit, meeting the requirement for serverless, managed file ingestion. AWS Glue can then be used to process the CSV files in S3, as it is a serverless ETL service suitable for batch processing large files. Option A (Kinesis Data Firehose) is for streaming data, not batch file transfers. Option B (Lambda) has a 15-minute timeout and is not ideal for large files up to 10 GB. Option E (Athena) is a query service, not a transfer or processing service.

Answer

Amazon Kinesis Data Firehose

Answer

AWS Lambda

Answer

Amazon Athena

Question 4

A company is ingesting IoT sensor data into Amazon Kinesis Data Streams. Each sensor sends a JSON payload every second. The data must be transformed and aggregated in real-time before being stored in Amazon DynamoDB. Which THREE services should be used together in the pipeline? (Choose THREE.)

Accepted Answer

AWS Lambda. Amazon Kinesis Data Streams (D) is the ingestion layer that captures the JSON payloads from IoT sensors in real-time, ensuring data is available for processing. AWS Lambda (A) can be used as a consumer of the stream to perform lightweight, per-record transformations, such as filtering or enriching the JSON payloads. Amazon Kinesis Data Analytics (B) is required for real-time aggregation and complex transformations using SQL or Apache Flink, enabling calculations like averages or counts per second before storing in DynamoDB. Together, these three services form a complete real-time pipeline: ingest with Kinesis Data Streams, transform/aggregate with Kinesis Data Analytics, and optionally further transform with Lambda before writing to DynamoDB.

Answer

Amazon S3

Answer

Amazon Kinesis Data Firehose

Question 5

A company wants to use AWS Glue to transform data stored in Amazon S3. The data is partitioned by date and includes both CSV and Parquet files. The transformation should be optimized for cost and performance. Which THREE actions should the data engineer take? (Choose THREE.)

Accepted Answer

Use partition pruning by filtering on the date column in the ETL script.. Options B, C, and E are correct. Partition pruning (B) reduces the amount of data scanned by filtering on the date column, lowering cost and improving performance. Job bookmarks (C) track processed data, preventing reprocessing and saving time and cost. Converting files to Parquet (E) reduces data size and enhances query performance due to its columnar format. Option A is incorrect because running a crawler before each job creates unnecessary overhead; the schema can be defined once or crawled periodically. Option D is incorrect because increasing DPUs to the maximum is costly and does not optimize performance; proper data partitioning and format are more effective.

Answer

Run a crawler to update the schema before each job run.

Answer

Increase the number of DPUs to the maximum allowed.

Question 6

Refer to the exhibit. An IAM policy is attached to an EC2 instance role that runs a data ingestion application. The application reads files from an S3 bucket 'data-lake-primary' and sends records to a Kinesis stream named 'clickstream'. The application is failing with an 'AccessDenied' error when trying to read from S3. What is the MOST likely cause?

Accepted Answer

The policy does not include the s3:ListBucket permission.. The application reads files from an S3 bucket, which requires both s3:GetObject (to read the object) and s3:ListBucket (to list objects in the bucket, typically needed for operations like listing or navigating the bucket). Without s3:ListBucket, the SDK may fail with an AccessDenied error when it tries to enumerate objects or validate the bucket path, even if s3:GetObject is present. Option D correctly identifies this missing permission as the most likely cause.

Answer

The actions are specified incorrectly; they should be s3:GetObject and s3:PutObject only.

Answer

The policy is not attached to the EC2 instance role.

Answer

The Kinesis stream name is incorrect.

Question 7

Refer to the exhibit. A data engineer runs the describe-stream command and sees this output. The application is writing records to the stream but is experiencing high write latency. The average record size is 50 KB, and the write rate is 1500 records per second. What is the MOST likely cause of the latency?

Accepted Answer

The stream does not have enough shards to handle the write throughput.. The stream does not have enough shards to handle the write throughput. Each Kinesis shard can ingest up to 1 MB/s or 1000 records per second (for up to 1 KB record size). With 2 shards, the total write capacity is 2 MB/s or 2000 records/s. However, the average record size is 50 KB, and the write rate is 1500 records/s. The throughput in MB/s is 1500 * 50 KB = 75 MB/s, which far exceeds the 2 MB/s total capacity. Therefore, the stream is under-provisioned, causing high write latency. Option A is incorrect because the region of the application does not directly cause high latency; cross-region latency might be a factor but not the most likely given the throughput metrics. Option B is incorrect because DynamoDB provisioned throughput is not relevant to Kinesis streams. Option C is incorrect because throttling due to a hot shard would occur if data is unevenly distributed, but the issue here is overall insufficient shard count given the high throughput demand.

Answer

The application is running in a different AWS region.

Answer

The application is exceeding the DynamoDB provisioned throughput.

Answer

The Kinesis stream is throttling the application because of a hot shard.

Question 8

Refer to the exhibit. A data engineer runs this AWS Glue Data Catalog DDL statement to create a table. The CSV files in 's3://my-bucket/sales/' use a pipe delimiter (|) instead of a comma. What change is needed to correctly read the data?

Accepted Answer

Change the 'field.delim' property to '|'.. The AWS Glue Data Catalog DDL statement uses the default 'field.delim' property, which expects comma-separated values. Since the CSV files use a pipe delimiter (|), the table will not parse rows correctly. Setting 'field.delim' to '|' in the SerDe properties tells the Hive-compatible SerDe to split on pipes instead of commas, enabling correct data ingestion.

Answer

Change the LOCATION to read from a subfolder.

Answer

Add a partition projection configuration.

Answer

Run a crawler to detect the schema automatically.

Question 9

A company wants to ingest streaming data from thousands of IoT devices into AWS for real-time analytics. The data volume is variable and can spike unpredictably. The solution must be serverless and minimize operational overhead. Which AWS service should be used for ingestion?

Accepted Answer

Use Amazon Kinesis Data Streams to ingest and process data in real time.. Amazon Kinesis Data Streams is the correct choice because it is a serverless, real-time data ingestion service designed to handle variable and unpredictable data volumes from thousands of sources. It provides durable, low-latency streaming with the ability to process data in real time using consumers like Lambda or Kinesis Data Analytics, meeting the requirements for real-time analytics and minimal operational overhead.

Answer

Use Amazon Kinesis Data Firehose to load streaming data directly into Amazon S3.

Answer

Use Amazon SQS to queue messages and process them in batches.

Answer

Use AWS IoT Core to ingest data and route it to Amazon DynamoDB.

Question 10

A data engineer needs to transfer 50 TB of historical data from an on-premises HDFS cluster to Amazon S3. The on-premises network has a 1 Gbps link to AWS. The transfer must complete within 5 days. Which solution is MOST cost-effective and meets the requirements?

Accepted Answer

Use AWS Snowball Edge to physically transfer the data.. AWS Snowball Edge is a physical device that can transfer large amounts of data faster than a network link, especially with a 1 Gbps link that would take about 4.6 days for 50 TB (theoretical max, but actual throughput will be lower due to overhead). Snowball Edge can transfer 50 TB in a few days and is cost-effective for large data volumes. Option A (Amazon S3 Transfer Acceleration) speeds up transfers but still limited by network bandwidth. Option B (AWS DataSync) is efficient for online transfers but may not meet the 5-day deadline over 1 Gbps. Option D (AWS Direct Connect) would require additional setup and cost, and still limited by the 1 Gbps link.

Answer

Use Amazon S3 Transfer Acceleration to speed up the transfer over the internet.

Answer

Use AWS DataSync to transfer the data over the existing network link.

Answer

Use AWS Direct Connect to establish a dedicated network connection.

Question 11

A company is ingesting streaming data from multiple sources using Amazon Kinesis Data Streams. The data is then processed by an AWS Lambda function that transforms the records and writes them to an Amazon S3 bucket. The Lambda function is failing intermittently with timeout errors. The average record size is 5 KB, and the shard count is 2. What is the MOST likely cause of the timeout errors?

Accepted Answer

The Lambda function timeout is set too low for the processing time required.. The Lambda function is timing out, indicating that the configured timeout is insufficient for the actual processing time. Lambda has a default timeout of 3 seconds, but it can be set from 1 second to 15 minutes. If the transformation logic or S3 write operation takes longer than the configured timeout, the function will fail with a timeout error. Option B is incorrect because the Kinesis data retention period (default 24 hours) affects data availability, not Lambda execution time. Option C is incorrect because reserved concurrency controls the number of concurrent invocations and can cause throttling, not timeouts. Option D is incorrect because with an average record size of 5 KB, even the default batch size of 100 records results in only 500 KB per invocation, well below the 6 MB payload limit.

Answer

The Kinesis data retention period is too short, causing data to be lost before processing.

Answer

The Lambda function's reserved concurrency is set too low, causing throttling.

Answer

The Lambda function is receiving too many records per invocation, exceeding the 6 MB payload limit.

Question 12

A company is using AWS Database Migration Service (DMS) to migrate a 2 TB MySQL database to Amazon Aurora MySQL. The migration is taking longer than expected. The source database is in a different AWS region. Which change would MOST likely improve the migration speed?

Accepted Answer

Increase the number of parallel tables being migrated.. Increasing the number of parallel tables allows DMS to migrate multiple tables simultaneously, leveraging available bandwidth and reducing overall migration time. This is particularly effective for large databases (2 TB) where serial migration would be slow. Option B is incorrect because Multi-AZ provides high availability, not performance improvement; it adds overhead and does not reduce network latency. Option A reduces resources, making migration slower. Option D disables binary logging, which is needed for ongoing replication and can cause data loss or require a full re-sync.

Answer

Use a smaller DMS replication instance to reduce costs.

Answer

Use a Multi-AZ deployment for the DMS replication instance in the target region.

Answer

Disable binary logging on the source MySQL database.

Question 13

A company is streaming data from an application to Amazon Kinesis Data Streams. The data must be transformed in real time and then stored in Amazon S3 in Parquet format. Which AWS service should be used for the transformation step?

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink.. Amazon Kinesis Data Analytics for Apache Flink is a serverless service that can run Apache Flink applications to perform real-time transformations on streaming data. Option A (Amazon Kinesis Data Firehose with a Lambda transformation) is primarily for loading data and has limitations for complex transformations. Option B (Amazon EMR running Apache Spark Streaming) can work but adds management overhead and is not the simplest managed service. Option D (AWS Lambda with a Kinesis trigger) is suitable for lightweight transformations but may hit execution time limits for complex or long-running transformations.

Answer

Amazon Kinesis Data Firehose with a Lambda transformation.

Answer

Amazon EMR running Apache Spark Streaming.

Answer

AWS Lambda with a Kinesis trigger.

Question 14

A company is using AWS Glue to run ETL jobs that process data from Amazon S3 and load it into Amazon Redshift. The jobs are failing with the error 'Unable to connect to Redshift cluster'. The Redshift cluster is in the same VPC as the Glue job. What is the MOST likely cause?

Accepted Answer

The Glue job is not configured to use the same VPC as the Redshift cluster.. The most likely cause is that the Glue job is not configured to use the same VPC as the Redshift cluster. By default, AWS Glue jobs run in a separate VPC managed by Glue. To connect to resources in a customer VPC (like a Redshift cluster), the Glue job must be explicitly associated with that VPC via a Glue connection or job configuration. Option C is correct. Option A is incorrect because even if the security group allows inbound traffic, the Glue job's network may not be routed to the Redshift cluster if it is in a different VPC. Option B is incorrect because the IAM role handles authentication, not network connectivity. Option D is incorrect because the Redshift cluster is in the same VPC, so public accessibility is not required; the error indicates a network connectivity issue, not a public access issue.

Answer

The Redshift cluster's security group does not allow inbound traffic from the Glue job's security group.

Answer

The IAM role associated with the Glue job does not have permission to access Redshift.

Answer

The Redshift cluster is not publicly accessible and Glue is trying to connect from outside the VPC.

Question 15

A company needs to ingest data from an on-premises Oracle database into Amazon S3 on a daily basis. The data volume is about 100 GB per day. Which AWS service is BEST suited for this task?

Accepted Answer

Use AWS Database Migration Service (DMS) to replicate data to S3.. AWS Database Migration Service (DMS) can continuously replicate data from Oracle to S3, and it supports full load and change data capture (CDC). Option A (AWS DataSync) is for file-based transfers, not database replication. Option B (Amazon Kinesis Data Firehose) is for streaming data, not database pull. Option D (AWS Glue) is for ETL but does not natively support continuous CDC from Oracle.

Answer

Use AWS DataSync to copy the database files to S3.

Answer

Use Amazon Kinesis Data Firehose with a database connector.

Answer

Use AWS Glue to extract data from Oracle and write to S3.