MLS-C01 Data Engineering Practice Test 14 — 15 Questions

Question 1

A data engineer is designing a data lake on Amazon S3. The data comes from various sources, including IoT devices, web logs, and transactional databases. The engineer needs to organize the data in a way that supports efficient querying using Amazon Athena and allows for easy management of access permissions. Which S3 bucket structure is the most appropriate?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a prefix structure like s3://bucket/source/year/month/day/.. A prefix structure of s3://bucket/source/year/month/day/ aligns with how Athena and Glue partition data, enabling partition pruning so queries scan only relevant prefixes. Grouping by source first also keeps IAM policies and lifecycle rules manageable per data source. This hierarchical layout matches the Hive-style partitioning Athena expects.

Answer

Store all data in a single prefix without any partitioning.

Answer

Store all data in separate S3 buckets for each source and date.

Answer

Use a prefix structure like s3://bucket/date/source/.

Question 2

A data engineer is tasked with building a pipeline to process streaming data from IoT devices. The devices send data in JSON format every second. The pipeline must aggregate data in 5-minute windows and store the results in Amazon S3. The engineer needs to handle late-arriving data (up to 1 hour) and ensure exactly-once semantics. Which combination of AWS services should they use?

Accepted Answer

Amazon Kinesis Data Streams for ingestion, Amazon Kinesis Data Analytics for windowed aggregations, and Amazon Kinesis Data Firehose to write to Amazon S3.. Amazon Kinesis Data Analytics (KDA) supports windowed aggregations with configurable allowed lateness (up to 1 hour) and exactly-once processing via Flink checkpoints. Kinesis Data Firehose reliably writes aggregated results to S3. Kinesis Data Streams provides durable ingestion for streaming JSON data. This combination directly addresses the requirements for late data handling and exactly-once semantics.

Answer

Amazon Kinesis Data Streams for ingestion, AWS Glue Streaming ETL for aggregation, and Amazon S3 for storage.

Answer

Amazon SQS for ingestion, AWS Lambda for aggregation, and Amazon S3 for storage.

Answer

Amazon Kinesis Data Streams for ingestion, Amazon Kinesis Data Firehose for transformation, and Amazon S3 for storage.

Question 3

A research institution is building a data lake to store genomics data. Each experiment generates multiple files totaling about 500 GB. The data is stored in Amazon S3 and needs to be processed by multiple machine learning (ML) training jobs running on Amazon SageMaker. The data has a high churn rate; after 30 days, most data becomes irrelevant and should be moved to Amazon S3 Glacier Deep Archive. The institution wants to minimize storage costs while maintaining data durability. Which S3 storage class should they use for the first 30 days?

Accepted Answer

Use S3 Intelligent-Tiering for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.. S3 Intelligent-Tiering automatically moves data between access tiers based on changing access patterns, optimizing costs without performance impact. Since the data has a high churn rate and becomes irrelevant after 30 days, Intelligent-Tiering will automatically transition objects to the most cost-effective tier, and a lifecycle policy can then move them to Glacier Deep Archive after 30 days. This minimizes storage costs while maintaining durability (11 9's) and availability.

Answer

Use S3 One Zone-IA for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.

Answer

Use S3 Standard for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.

Answer

Use S3 Glacier Instant Retrieval for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.

Question 4

A company uses AWS Glue crawlers to populate the AWS Glue Data Catalog from Amazon S3. The data is partitioned by year/month/day/hour. The crawler runs every hour and adds new partitions. However, the data engineer notices that the crawler is taking longer to run as the number of partitions grows, and sometimes it misses new partitions. What is the most cost-effective and reliable way to address this?

Accepted Answer

Enable the crawler's partition index feature.. Enabling the crawler's partition index feature allows AWS Glue to quickly find new partitions without re-scanning the entire table, reducing runtime and improving reliability. Option B is incorrect because manually adding partitions is error-prone and does not scale. Option C is incorrect because the MSCK REPAIR TABLE command is a manual step that does not automate the process. Option D is incorrect because increasing the crawler's frequency does not address the underlying issue of scanning overhead and may lead to resource contention.

Answer

Manually add new partitions using ALTER TABLE ADD PARTITION in Athena.

Answer

Use the Athena MSCK REPAIR TABLE command after the crawler runs.

Answer

Increase the crawler's schedule to run every 30 minutes.

Question 5

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a 100 Mbps internet connection and the data must be transferred within 5 days. Which AWS service is best suited for this task?

Accepted Answer

AWS Snowball Edge. AWS Snowball Edge is a petabyte-scale data transport solution that uses secure physical devices to transfer large amounts of data into and out of AWS. With 50 TB of data and only a 100 Mbps internet connection, transferring over the internet would take over 46 days (50 TB = 400,000 Mb; at 100 Mbps, ~4,000,000 seconds ≈ 46 days), far exceeding the 5-day requirement. Snowball Edge is designed for such scenarios, enabling offline transfer within days.

Answer

AWS DataSync

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Storage Gateway

Question 6

A company is migrating on-premises data to AWS. The data includes both structured and unstructured files, totaling 200 TB. The company has a 1 Gbps dedicated network connection to AWS. They want to minimize migration time and cost. Which TWO AWS services or features should they use together? (Choose two.)

Accepted Answer

AWS Snowball Edge. AWS Snowball Edge (A) is correct because moving 200 TB over a 1 Gbps link would take roughly 18-20 days of continuous transfer, whereas shipping multiple Snowball Edge devices physically transfers the data offline and dramatically shortens the migration window while avoiding sustained data-transfer charges. AWS DataSync (D) is correct because it is the managed service designed to automate and accelerate online data movement into AWS (e.g., to Amazon S3 or Amazon EFS), handling scheduling, integrity verification, and incremental transfers for both structured and unstructured files, which complements Snowball for the remaining or ongoing data. Amazon S3 Transfer Acceleration (B) is not appropriate as a primary solution because it only speeds up transfers to S3 over the public internet using edge locations and does not address the 200 TB volume or cost concerns. AWS Glue (C) is a serverless ETL/catalog service, not a data migration tool, so it does not move the files. AWS Direct Connect (E) is already in place (the 1 Gbps dedicated connection) and is not a separate service to add for this migration.

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Glue

Answer

AWS Direct Connect

Question 7

A data engineering team is designing a streaming data pipeline that ingests 10,000 events per second. Each event is 2 KB. The pipeline must process events with a latency of less than 1 second. The team is considering using Amazon Kinesis Data Streams with 10 shards. Which TWO additional configurations should the team implement to meet the latency requirement? (Choose two.)

Accepted Answer

Enable auto-scaling of shards based on throughput.. Correct options: D and E. Enhanced fan-out (E) provides dedicated read throughput of 2 MB/second per shard per consumer, eliminating contention and ensuring low latency (<1 second) even with multiple consumers. Auto-scaling shards (D) dynamically adjusts shard count based on incoming throughput, which is critical because the current 10 shards can only ingest 10 MB/sec (1 MB/sec write per shard) but the input is 20 MB/sec (10,000 events × 2 KB each). Option A (record aggregation) reduces PUT request count but does not affect read latency. Option B (Kinesis Firehose to S3) is for data delivery, not latency. Option C (increasing shards to 100) provides excess capacity but is costly and less efficient than auto-scaling.

Answer

Use record aggregation to reduce the number of records.

Answer

Configure Kinesis Data Firehose to deliver data to Amazon S3.

Answer

Increase the number of shards to 100.

Question 8

A company uses AWS Glue to run ETL jobs. The data engineer wants to monitor job performance and troubleshoot failures. Which THREE AWS services or features should they use together? (Choose three.)

Accepted Answer

AWS Glue job bookmarks. AWS Glue job bookmarks (A) are the correct feature for tracking which data has already been processed across job runs, which is essential for monitoring incremental ETL progress and preventing reprocessing during troubleshooting. Amazon CloudWatch Logs (D) captures the detailed log output from Glue job runs, including error messages and stack traces, making it the primary tool for diagnosing why a job failed. Amazon CloudWatch metrics (E) provides the numeric performance indicators for Glue jobs, such as job run duration, success/failure counts, and DPU usage, which are needed to monitor performance trends. Amazon S3 Event Notifications (B) only trigger actions on object-level events and do not provide job monitoring or failure diagnostics, and Amazon Athena (C) is a query service for data in S3, not a tool for monitoring or troubleshooting Glue job execution.

Answer

Amazon S3 Event Notifications

Answer

Amazon Athena

Question 9

A data engineer is designing a data pipeline that uses Amazon Kinesis Data Streams to ingest real-time transaction data. The data must be processed in near real-time and stored in Amazon S3 for long-term analytics. The engineer wants to ensure data durability and exactly-once processing semantics. Which TWO actions should the engineer take? (Choose two.)

Accepted Answer

Use AWS Glue streaming ETL with checkpointing.. Option B is correct because AWS Glue streaming ETL natively consumes Kinesis Data Streams and uses job checkpointing (bookmarks) to track processed records, so on restart or retry it resumes from the last committed position and avoids reprocessing, which is how exactly-once semantics are achieved in the Glue/Kinesis integration. Option D is correct because an AWS Lambda function with a Kinesis event source mapping processes records in order per shard and only advances the shard iterator after successful invocation; configuring the event source mapping's retry behavior (e.g., RetryAttempts) controls failure handling so that failed batches are not silently skipped and duplicate processing is minimized, supporting durable, near-real-time delivery into S3. Option A is not correct because the Kinesis Producer Library does not provide exactly-once delivery to Kinesis Data Streams; it offers aggregation and retry configuration, but Kinesis itself provides at-least-once delivery, so duplicates can still occur. Option C is not correct because Kinesis Data Firehose does not expose an 'exactly-once delivery' setting; it provides at-least-once delivery with S3 backup and buffering options. Option E is not correct because Amazon SQS is a separate queueing service, not a Kinesis Data Streams consumer, and using it as the downstream event source does not by itself deliver exactly-once processing for the Kinesis pipeline.

Answer

Use the Kinesis Producer Library (KPL) with exactly-once delivery.

Answer

Enable exactly-once delivery on Kinesis Data Firehose.

Answer

Use Amazon SQS as the event source for downstream processing.

Question 10

A company runs an e-commerce platform that generates clickstream data in real-time. The data is ingested into Amazon Kinesis Data Streams (100 shards) and processed by AWS Lambda functions, which aggregate data in 1-minute windows and write the results to Amazon S3. The Lambda functions are triggered by the Kinesis stream using the event source mapping. Recently, the company noticed that some records are being processed multiple times, leading to duplicate data in S3. The Lambda function is idempotent, but the duplicates are causing downstream issues. The Lambda function's concurrency limit is 1000, and the batch size is 100. The average processing time per record is 200 ms. What is the most likely cause of the duplicates, and how should it be fixed?

Accepted Answer

Ensure the Lambda function is idempotent and uses the sequence number to deduplicate records.. Lambda functions process records from Kinesis in batches. If the function fails (e.g., due to timeout or error), the entire batch is retried, causing duplicates if some records were already partially processed. To avoid duplicates, the function should be idempotent and should not commit partial results. Option A is wrong because the concurrency is sufficient. Option C is wrong because increasing batch size increases the risk of partial failure. Option D is wrong because a FIFO queue does not integrate with Kinesis.

Answer

Increase the Lambda concurrency limit to 2000 to handle the load.

Answer

Decrease the batch size to 10 to reduce the impact of failures.

Answer

Use Amazon SQS FIFO queue as a buffer between Kinesis and Lambda to guarantee exactly-once processing.

Question 11

A data engineer is building a data pipeline that ingests streaming data from Amazon Kinesis Data Streams, transforms the data using AWS Lambda, and stores the results in Amazon S3. The engineer needs to ensure that each record is processed exactly once and in order. Which TWO approaches should the engineer consider? (Choose TWO.)

Accepted Answer

Configure the Lambda function with a reserved concurrency of 1 and a batch size of 1 to process records sequentially.. To achieve exactly-once processing and in-order delivery from Kinesis Data Streams with Lambda, the engineer should consider:
- Option A: Configuring reserved concurrency of 1 and batch size of 1 ensures that each Lambda invocation processes one record at a time and that records within a shard are processed sequentially, maintaining ordering.
- Option B: Using the Kinesis Producer Library (KPL) (or relying on the sequence number already assigned to each record) allows the consumer Lambda to implement idempotent processing by checking the sequence number against a deduplication store (e.g., DynamoDB with conditional writes), thereby achieving exactly-once semantics.
Combining these approaches helps meet both requirements.

Option C is not correct because integrating SQS FIFO queues between Kinesis and Lambda adds unnecessary complexity and does not directly preserve the ordering across shards without additional mapping; also, Kinesis Data Streams does not natively fan-out to SQS FIFO.
Option D is incorrect because S3 Event Notifications are asynchronous and do not address ordering or exactly-once processing in the pipeline.
Option E is incorrect because Kinesis Data Firehose is designed for near-real-time delivery and can introduce duplicate records; it does not guarantee exactly-once processing.

Answer

Use Amazon SQS FIFO queues to decouple Kinesis and Lambda, ensuring ordering and exactly-once delivery.

Answer

Enable S3 Event Notifications to trigger Lambda for each object.

Answer

Use Kinesis Data Firehose to buffer and deliver data to S3, then use Lambda to process.

Question 12

A company runs a data lake on Amazon S3 with AWS Glue for ETL. The data science team needs to train machine learning models on historical data, but they are concerned about data quality issues such as missing values, duplicates, and outliers. The team wants to build a data quality monitoring solution that automatically detects anomalies and alerts the data engineering team. Which THREE steps should the team take to implement this solution? (Choose THREE.)

Accepted Answer

Use AWS Glue DataBrew to create data quality rules that check for missing values, duplicates, and outliers, and schedule them to run regularly.. AWS Glue DataBrew provides built-in data quality checks and profiling. Option C is correct because Amazon CloudWatch alarms can be set on data quality metrics to trigger SNS notifications when thresholds are breached. Option D is correct because the Deequ library can be implemented on Amazon EMR to compute data quality metrics and store them in CloudWatch. Option B is incorrect because Amazon Kinesis Data Analytics is used for real-time streaming analytics, not for batch data quality checks on historical data. Option E is incorrect because Amazon SageMaker Processing jobs are for model training and data processing, not specifically for data quality monitoring.

Answer

Use Amazon Kinesis Data Analytics to continuously monitor streaming data for data quality issues.

Answer

Use Amazon SageMaker Processing jobs to run custom data quality scripts and store results in SageMaker Experiments.

Question 13

A data engineering team is building a real-time data pipeline using Amazon Kinesis Data Streams with AWS Lambda for processing. The pipeline ingests clickstream data from a mobile app. The team notices that occasionally, a Lambda function fails due to a transient error, and the failed record is not retried, leading to data loss. The Lambda function is configured with a batch size of 100 and a maximum retry count of 0. The team wants to ensure that all records are processed successfully, even if transient failures occur. They also want to minimize the impact of poison pill records that could block processing. Which combination of actions should the team take to address this issue?

Accepted Answer

Set the maximum retry count to 5, configure an on-failure destination Amazon SQS queue, and set up a dead-letter queue on that SQS queue for poison pills.. It addresses both transient failures and poison pill records. Setting the maximum retry count to 5 ensures that transient errors are retried, reducing data loss. Configuring an on-failure destination (Amazon SQS) captures records that fail after all retries, preventing data loss. Adding a dead-letter queue on that SQS queue isolates poison pill records, allowing the main stream to continue processing without blockage.

Answer

Set the maximum retry count to 5 and configure a dead-letter queue on the Lambda function to capture failed records after retries.

Answer

Switch to using Amazon Kinesis Data Firehose to buffer data and use AWS Lambda for transformation with built-in retry logic.

Answer

Reduce the batch size to 1 and increase the Lambda function timeout to handle transient errors.

Question 14

A company is building a data pipeline to process streaming data from IoT devices. The data is ingested via Amazon Kinesis Data Streams. Each record is about 1 KB. The company wants to use AWS Lambda for real-time transformations and then store the results in Amazon DynamoDB. The expected throughput is 10,000 records per second. The Lambda function currently runs in about 200 ms. The company is concerned about Lambda concurrency limits and wants to ensure there are no throttling errors. The default concurrency limit for Lambda is 1,000. Which approach should the team take to handle the expected throughput without throttling?

Accepted Answer

Increase the number of shards in the Kinesis Data Stream to 10 and set the Lambda batch size to 100.. With 10,000 records per second at 1 KB each, the stream needs enough shards to handle the throughput. Each Kinesis shard supports 1 MB/s or 1,000 records/s ingress, so 10 shards are required for 10,000 records/s. Lambda's Kinesis integration processes one batch per shard invocation, and with a batch size of 100, each invocation handles 100 records. At 200 ms per invocation, each shard can process roughly 5 batches per second (100 records × 5 = 500 records/s per shard), and 10 shards yield 5,000 records/s — but with Lambda's default 1,000 concurrent executions and the shard-level concurrency model, this configuration keeps concurrency well within limits while matching shard throughput to the incoming record rate.

Answer

Increase the Lambda function memory to 3,000 MB to reduce the execution time below 100 ms.

Answer

Use Amazon Kinesis Data Firehose instead of Lambda to load data directly into DynamoDB.

Answer

Reduce the Lambda batch size to 10 so that each invocation processes fewer records, reducing the time per invocation.

Question 15

A data engineer is setting up a data lake on Amazon S3 for a large retail company. The data includes customer transactions, inventory, and web logs. The company wants to use AWS Glue for ETL and Amazon Athena for ad-hoc queries. The data is partitioned by year, month, day, and hour. The engineer notices that Athena queries are slow and often scan large amounts of data even when only a specific hour is needed. The engineer has already enabled partitioning and used columnar formats like Parquet. What additional step should the engineer take to optimize query performance and reduce data scanned?

Accepted Answer

Use a coarser partition layout, such as partitioning only by date, and leverage Hive-style partitioning with AWS Glue Crawlers to avoid excessive small files.. The issue is that the current partitioning scheme (year/month/day/hour) creates too many small files, which increases overhead for Athena's query planning and listing operations. Coarsening the partition layout to just date reduces the number of partitions and file count, while still allowing efficient pruning for most queries. Hive-style partitioning with AWS Glue Crawlers ensures the partitions are properly registered in the Glue Data Catalog, enabling Athena to skip irrelevant data. This balances granularity and performance, reducing data scanned and improving query speed.

Answer

Convert the Parquet files to CSV format to reduce the overhead of columnar storage and improve compression.

Answer

Use S3 Select to push down filters to S3, reducing the amount of data scanned by Athena.

Answer

Increase the granularity of partitioning to include minute-level partitions to further limit data scanned.