DEA-C01 Data Ingestion and Transformation Practice Test 7 — 15 Questions

Question 1

A company uses Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data is in JSON format and each record is about 2 KB. The delivery stream is configured to buffer data for 60 seconds or 5 MB, whichever comes first. The team notices that the S3 objects are very small (around 1 MB) and numerous, causing high costs due to S3 PUT requests. Which configuration change should the team make to reduce the number of S3 objects?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the buffer size to 50 MB and the buffer interval to 300 seconds.. Increasing the buffer size to 50 MB and the buffer interval to 300 seconds allows Kinesis Data Firehose to accumulate more data before writing to S3, resulting in fewer, larger objects. The current configuration triggers a write every 60 seconds or when 5 MB is buffered, but since each record is only 2 KB, the 5 MB threshold is rarely met, causing frequent small writes. By raising both thresholds, the delivery stream will buffer more records and write larger objects, reducing the number of S3 PUT requests and associated costs.

Answer

Enable compression (GZIP) on the delivery stream.

Answer

Reduce the buffer interval to 30 seconds and keep buffer size at 5 MB.

Answer

Switch from Kinesis Data Firehose to Amazon Kinesis Data Streams and use a Lambda function to write to S3.

Question 2

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a slow internet connection (100 Mbps). The data must be transferred within 2 weeks. Which service should the engineer recommend?

Accepted Answer

AWS Snowball Edge. B is correct because transferring 50 TB over a 100 Mbps connection would take approximately 48 days (50 TB * 1024 GB/TB * 8 bits/byte / (100 Mbps * 86400 seconds/day)), far exceeding the 2-week deadline. AWS Snowball Edge is a physical appliance that can be loaded with data on-premises and shipped to AWS, bypassing the slow network entirely and completing the transfer within days.

Answer

AWS DataSync

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue ETL job with JDBC connection to Hadoop

Question 3

A company is using Amazon Kinesis Data Streams to ingest clickstream data from a website. The data is consumed by an AWS Lambda function that enriches records and writes to Amazon S3. The Lambda function is experiencing high error rates due to records exceeding the 256 KB payload limit. Which TWO actions should the team take to resolve this issue?

Accepted Answer

Enable compression on the producer side before sending records to Kinesis.. Enabling compression on the producer side reduces the size of each record before it is sent to Kinesis Data Streams, directly addressing the 256 KB payload limit. Option C is correct because the Kinesis Producer Library (KPL) aggregates multiple small records into a single larger record, which is then stored as one Kinesis record, reducing the number of records that exceed the limit and improving throughput.

Answer

Increase the Lambda function timeout.

Answer

Switch from Kinesis Data Streams to Kinesis Data Firehose.

Answer

Increase the number of shards in the Kinesis stream.

Question 4

A data engineering team is building a data lake on Amazon S3. They need to ingest data from multiple sources: (1) streaming IoT data, (2) daily CSV exports from an on-premises system via SFTP, and (3) change data capture (CDC) from an Amazon Aurora database. Which THREE services should the team use to ingest these data sources?

Accepted Answer

Amazon Kinesis Data Streams for IoT data ingestion.. Amazon Kinesis Data Streams is purpose-built for real-time streaming data ingestion, making it ideal for IoT data that arrives continuously. It can capture and store data streams for processing by consumers like Kinesis Data Analytics or Lambda, providing low-latency ingestion and durable storage.

Answer

AWS Glue ETL for CDC from Aurora.

Answer

Amazon EMR for daily CSV ingestion.

Question 5

A data engineer is ingesting streaming data from an IoT fleet into Amazon S3 using Amazon Kinesis Data Firehose. The data arrives as JSON, but the downstream analytics require Parquet format. Which Firehose transformation should the engineer configure?

Accepted Answer

Configure a Lambda function as a data transformation in Firehose to convert JSON to Parquet.. Amazon Kinesis Data Firehose can invoke an AWS Lambda function as a data transformation step to convert incoming JSON records to Parquet format before delivery to S3. This is the native, serverless way to perform record-level format conversion within the Firehose delivery stream, ensuring downstream analytics tools can directly query the Parquet data without additional processing.

Answer

Use an S3 lifecycle policy to convert JSON to Parquet.

Answer

Use S3 Batch Operations to convert existing JSON objects to Parquet.

Answer

Use Kinesis Data Analytics to convert the stream to Parquet before writing to S3.

Question 6

A company uses AWS Glue ETL to process data from Amazon S3 and write results to Amazon Redshift. The job fails with a memory error when processing large files. Which action should the data engineer take to resolve this issue?

Accepted Answer

Increase the number of DPUs allocated to the Glue job.. Increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job provides more memory and compute resources, which directly addresses the out-of-memory error when processing large files. Glue jobs run on Apache Spark, and insufficient DPUs can cause executors to run out of memory during shuffle or aggregation operations on large datasets.

Answer

Reduce the number of partitions in the Glue job.

Answer

Switch to a smaller instance type in the Glue job configuration.

Answer

Use S3 Select to filter columns before reading into Glue.

Question 7

A company uses AWS Database Migration Service (DMS) to continuously replicate data from an on-premises Oracle database to Amazon S3. The data is stored as CSV files. The downstream team requires the data to be in Apache Parquet format. Which change should the data engineer make to the DMS task?

Accepted Answer

Modify the DMS task to use Apache Parquet as the target table preparation mode.. AWS DMS supports specifying Apache Parquet as the target data format for Amazon S3 targets directly within the task configuration. By setting the 'Data format' to 'Parquet' in the S3 target endpoint or task settings, DMS automatically converts the replicated data into Parquet files, eliminating the need for post-processing. This is the most efficient and native approach to meet the downstream team's requirement.

Answer

Add an S3 lifecycle rule to convert CSV to Parquet.

Answer

Change the DMS task to use full load instead of continuous replication.

Answer

Configure a Lambda function to transform data after DMS writes to S3.

Question 8

A data engineer is building a data ingestion pipeline that reads JSON files from Amazon S3 and loads them into an Amazon Redshift table using COPY commands. The files are gzip compressed and contain nested JSON. The engineer wants to minimize transformation steps. Which approach should the engineer use?

Accepted Answer

Use the COPY command with the 'auto' option to ingest JSON directly.. The COPY command with the 'auto' option can directly ingest gzip-compressed JSON files from S3 into Redshift, automatically inferring the schema and handling nested structures without requiring intermediate transformation steps. This minimizes transformation steps by leveraging Redshift's native JSON parsing capability, which supports both 'auto' and 'jsonpaths' options for nested data.

Answer

Use Amazon Athena to query the JSON and INSERT INTO Redshift.

Answer

Use Kinesis Data Firehose to transform and load into Redshift.

Answer

Use AWS Glue ETL to flatten the JSON and write to S3 as CSV, then COPY from CSV.

Question 9

A company is migrating its on-premises data warehouse to Amazon Redshift. The daily batch load from the source database takes 6 hours using a single-node Redshift cluster. The engineer needs to reduce load time to under 2 hours without increasing cost significantly. Which strategy should the engineer adopt?

Accepted Answer

Increase the number of nodes in the Redshift cluster and use parallel COPY from multiple files.. Increasing the number of nodes in the Redshift cluster provides more compute and I/O capacity, and using parallel COPY from multiple files allows Redshift to automatically split the load across the node slices, dramatically reducing load time. This approach scales performance linearly with the number of nodes, enabling the engineer to meet the sub-2-hour target without significantly increasing cost if the cluster is sized appropriately.

Answer

Use COPY with compression (gzip) to reduce data volume.

Answer

Use a VPC endpoint to improve network throughput to S3.

Answer

Change the table distribution style to EVEN to distribute data evenly.

Question 10

A data engineer needs to ingest data from an external HTTP API into Amazon S3. The API returns JSON data for a list of users, updated hourly. The engineer wants to use a serverless solution with minimal operational overhead. Which AWS service should the engineer use?

Accepted Answer

Amazon AppFlow with an HTTP connector on a scheduled flow.. Amazon AppFlow with an HTTP connector on a scheduled flow is the correct choice because it provides a fully managed, serverless integration that directly connects to external HTTP APIs, retrieves JSON data, and writes it to Amazon S3 on a scheduled basis (e.g., hourly) without requiring any custom code or infrastructure management. This minimizes operational overhead while meeting the ingestion requirements.

Answer

Amazon Kinesis Data Firehose with a custom HTTP endpoint.

Answer

AWS Lambda function triggered by CloudWatch Events.

Answer

AWS Glue ETL job triggered by EventBridge.

Question 11

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that is failing to deliver data to an Amazon S3 bucket. The stream is configured with a Lambda transformation function. The CloudWatch logs show that the Lambda function is timing out. Which action should the engineer take to resolve the issue?

Accepted Answer

Increase the Lambda function timeout setting.. The CloudWatch logs indicate the Lambda function is timing out. The default Lambda timeout for a Firehose transformation is 60 seconds, and if the function's processing exceeds this limit, it will fail. Increasing the Lambda function timeout setting (Option B) directly addresses this by allowing the function more time to complete its execution before being terminated.

Answer

Reduce the Firehose buffer interval.

Answer

Decrease the Lambda function's batch size in Firehose.

Answer

Increase the memory allocated to the Lambda function.

Question 12

A company uses AWS Glue to run ETL jobs that process data from Amazon RDS for MySQL and load it into Amazon S3. The job runs daily and processes incremental changes using the JDBC connection. Recently, the job has been failing with a 'Communications link failure' error. The RDS instance is in a private subnet. Which step should the engineer take first to diagnose the issue?

Accepted Answer

Check the security group and network ACL rules for the RDS instance and the Glue connection.. The 'Communications link failure' error typically indicates a network connectivity issue between AWS Glue and the RDS instance. Since the RDS instance is in a private subnet, the Glue job must be able to reach it via a VPC endpoint or a Glue connection that uses network configuration. Checking the security group (inbound rules for the RDS instance allowing traffic from Glue's elastic network interfaces) and network ACLs (ensuring ephemeral ports are open) is the first logical step to diagnose connectivity.

Answer

Verify that the IAM role used by Glue has the correct permissions to access RDS.

Answer

Change the Glue job type from Spark to Python shell.

Answer

Check that the JDBC driver is compatible with the Glue version.

Question 13

A data engineer is designing a data ingestion pipeline for streaming social media data. The data must be ingested with low latency (seconds) and stored in Amazon S3 for long-term analytics. The engineer also needs to perform real-time aggregations. Which TWO services should the engineer use? (Choose two.)

Accepted Answer

Amazon Kinesis Data Analytics. Amazon Kinesis Data Analytics is correct because it enables real-time processing and aggregation of streaming data using SQL or Apache Flink, allowing the engineer to compute metrics like counts or averages on the fly. Combined with Kinesis Data Streams as the ingestion layer, it provides sub-second latency for both ingestion and analytics, meeting the low-latency requirement.

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue ETL

Answer

Amazon S3

Question 14

A company is building a data lake on Amazon S3. The data sources include relational databases, streaming data, and log files. The data engineer needs to ensure that the data ingestion pipeline can handle schema evolution, support both batch and streaming, and provide a unified metadata catalog. Which THREE services should the engineer use? (Choose three.)

Accepted Answer

AWS Glue. AWS Glue is correct because it provides a unified metadata catalog (the AWS Glue Data Catalog) that stores schema information for data stored in Amazon S3. It supports schema evolution by allowing you to update the catalog schema as data formats change, and it integrates with both batch (AWS Glue ETL jobs) and streaming (AWS Glue Streaming ETL) ingestion pipelines, making it the central service for metadata management in a data lake.

Answer

Amazon DynamoDB

Answer

Amazon Athena

Question 15

A data engineer is troubleshooting a slow-running AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job processes 500 GB of CSV data daily. The engineer wants to improve performance. Which THREE actions should the engineer take? (Choose three.)

Accepted Answer

Use a JDBC connection with a higher batch size for writing to Redshift.. Increasing the JDBC batch size for the Redshift connection reduces the number of network round trips and improves write throughput. The Glue JDBC connector batches rows into a single INSERT statement; a larger batch size (e.g., 1000 instead of the default 100) allows more rows per commit, reducing overhead and speeding up the write phase.

Answer

Switch to a single-node Redshift cluster to reduce latency.

Answer

Reduce the number of input files by combining them into larger files.