MLS-C01 Data Engineering Practice Test 7 — 15 Questions

Question 1

A company uses AWS Lambda to process events from Amazon S3. The Lambda function transforms the data and writes results to another S3 bucket. Recently, the function has been failing due to timeout errors when processing large files. Which solution should the data engineer implement?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use S3 Batch Operations with a Lambda function to process objects. S3 Batch Operations is designed to handle large-scale object processing by invoking a Lambda function asynchronously for each object, bypassing the synchronous invocation limits of S3 event notifications. This allows processing of large files without hitting Lambda's 15-minute timeout or memory constraints, as each object is processed independently and the operation can scale to billions of objects.

Answer

Increase the Lambda function memory and timeout limit

Answer

Increase the Lambda timeout to 15 minutes

Answer

Use Amazon SQS to queue the events and process them in batches

Question 2

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a 1 Gbps internet connection and wants to complete the transfer within 5 days. What is the MOST cost-effective and reliable solution?

Accepted Answer

Use AWS Snowball Edge device to physically ship the data. AWS Snowball Edge is the most cost-effective and reliable solution because transferring 50 TB over a 1 Gbps internet connection would take approximately 5.5 days under ideal conditions (50 TB * 1024 GB/TB * 8 bits/byte / (1 Gbps * 86400 seconds/day) ≈ 4.74 days, but real-world overhead, congestion, and retransmissions push it beyond 5 days). Snowball Edge provides a physical appliance that can be shipped, avoiding network bandwidth limitations entirely, and is designed for large-scale data transfers where internet speeds are insufficient.

Answer

Use S3 multipart upload over the internet

Answer

Set up AWS Direct Connect and transfer over the dedicated line

Answer

Use S3 Transfer Acceleration to speed up the transfer

Question 3

A company is using Amazon Kinesis Data Streams to ingest clickstream data. The data is consumed by a fleet of EC2 instances running a custom consumer application. The consumer is falling behind and the shard iterator age is increasing. Which TWO actions should the data engineer take to improve consumer performance? (Choose TWO.)

Accepted Answer

Increase the number of shards in the stream. Option A is correct because increasing the number of shards horizontally scales the stream's read capacity: each shard supports up to 2 MiB/sec of read throughput (5 read transactions/sec), so adding shards lets the consumer fleet parallelize reads across more shards and reduce the growing shard iterator age. Option D is correct because enhanced fan-out gives each registered consumer a dedicated 2 MiB/sec read throughput per shard over HTTP/2, instead of sharing the shard's 2 MiB/sec limit with all consumers, which directly improves consumer throughput and lowers iterator age. Option B is wrong because the data retention period (default 24 hours, up to 365 days) only controls how long records remain accessible; shortening it does not increase read throughput and risks data loss. Option C is wrong because replacing the EC2 consumer with Lambda does not inherently raise read throughput and can even worsen lag due to concurrency and batch limits. Option E is wrong because the KCL is a processing library that helps manage shard leases and checkpointing, but it does not by itself increase the stream's read capacity or fix a throughput bottleneck.

Answer

Decrease the data retention period

Answer

Use an AWS Lambda function to process the data

Answer

Switch to the Kinesis Client Library (KCL)

Question 4

A data engineer is designing a data pipeline to process streaming data from Amazon Kinesis Data Streams and store the results in Amazon S3 in Parquet format. The data must be available for querying in Amazon Athena within minutes of arrival. Which THREE services should be used together? (Choose THREE.)

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose (C) is correct because it can ingest streaming data from Kinesis Data Streams and deliver it directly to Amazon S3, with built-in support for format conversion to Parquet using AWS Glue Data Catalog schemas. Amazon Kinesis Data Analytics (D) is correct because it provides real-time SQL or Apache Flink processing of the streaming data before delivery, enabling transformations and aggregations within minutes. AWS Glue (E) is correct because the Glue Data Catalog stores the table metadata and schema that Athena needs to query the Parquet data in S3, and Glue ETL can further catalog or transform the data. Amazon EMR (A) is not required because the scenario does not demand custom Hadoop/Spark cluster processing, and Firehose plus Kinesis Data Analytics already handle the streaming transformation. Amazon Redshift (B) is not appropriate because the target storage is Amazon S3 in Parquet format for Athena querying, not a Redshift data warehouse.

Answer

Amazon EMR

Answer

Amazon Redshift

Question 5

A data engineer is troubleshooting an AWS Glue job that reads from an S3 bucket and writes to another S3 bucket. The job fails with an 'Access Denied' error when trying to write to the output bucket. The IAM policy attached to the Glue service role is shown. What is the MOST likely cause of the failure?

Accepted Answer

The output bucket is not listed in the Resource of the IAM policy. The IAM policy attached to the Glue service role explicitly lists the output bucket in the Resource field. If the output bucket is not listed, the Glue job will receive an 'Access Denied' error when attempting to write to it, because the policy does not grant the necessary s3:PutObject permission for that bucket. This is the most direct cause of the failure.

Answer

The user who runs the job does not have S3 permissions

Answer

The Glue job role does not have permissions to start a job run

Answer

The S3 bucket policy denies access to the Glue service

Question 6

A company runs a data pipeline using AWS Glue ETL jobs that process about 10 TB of data daily from Amazon S3. The jobs are triggered by a schedule and write results to a separate S3 bucket. Recently, the jobs have been taking longer to complete, and the data engineering team has observed that the number of files in the source bucket has increased significantly, from thousands to millions of small files (each about 100 KB). The Glue jobs are configured to use the 'Group Files' option, but performance is still poor. The team needs to improve the job performance without changing the source data generation process. Which course of action should the team take?

Accepted Answer

Create a separate Glue job that runs before the main job to consolidate small files into larger ones in the source bucket. Creating a separate Glue job to consolidate small files into larger ones before the main job is the best approach. This reduces the number of files, improving read performance and reducing overhead in the main ETL job, without altering the source data generation process.

Answer

Increase the number of DPUs allocated to the existing Glue job

Answer

Switch the ETL processing to Amazon EMR with Spark

Answer

Use AWS Lambda to pre-process the files and combine them

Question 7

An e-commerce company uses Amazon Kinesis Data Firehose to deliver clickstream data to an Amazon S3 bucket. The data is then queried using Amazon Athena. The marketing team wants to run daily reports that aggregate click events by product ID. However, the reports are slow because Athena scans the entire dataset each time. The data is partitioned by date (e.g., s3://bucket/clickstream/2023/01/01/). The product ID is a column within the data. The data engineering team wants to improve query performance without moving the data to another service. Which approach should the team take?

Accepted Answer

Repartition the data by product ID in addition to date. Repartition the data by product ID in addition to date. This adds a partition level for product ID, so queries that filter on product ID will only scan the relevant partitions. Option A (convert to Parquet) reduces data scanned due to columnar storage and compression, but without partition pruning on product ID, Athena would still scan all partitions for each query. Option B (Redshift Spectrum) would still require scanning data, and involves additional service complexity. Option C (create a view) does not change physical storage; it only provides a logical filter, but Athena still scans all underlying data. Therefore, repartitioning by product ID provides the most direct improvement for queries filtering by product ID.

Answer

Convert the data from JSON to Parquet format

Answer

Use Amazon Redshift Spectrum to query the data

Answer

Create a view in Athena that filters by product ID

Question 8

A startup is building a data pipeline that ingests data from multiple sources into an Amazon S3 data lake. The data includes CSV files from legacy systems, JSON from web APIs, and Avro from mobile apps. The data must be transformed into Parquet format and cataloged for querying with Amazon Athena. The pipeline must be serverless and minimize operational overhead. The team has decided to use AWS Glue for ETL and cataloging. However, they are concerned about the cost of running Glue jobs continuously. The data arrives in small batches every 10 minutes. Which approach should the team use to minimize cost while meeting the requirements?

Accepted Answer

Use scheduled Glue jobs to process the data every hour, consolidating multiple batches. Using scheduled Glue jobs every hour to process accumulated data reduces the number of job runs and associated costs, while still providing near-real-time processing (within the hour). Option A is wrong because Lambda functions have limited execution time and memory, making them unsuitable for large-scale transformations. Option B is wrong because Kinesis Data Firehose can directly deliver streaming data to S3, but it does not handle all source formats natively (e.g., CSV, Avro) and additional transformation may be needed. Option D is wrong because a single daily Glue job introduces too much latency for batch arrivals every 10 minutes.

Answer

Use AWS Lambda functions to transform each file upon arrival and store as Parquet

Answer

Use Amazon Kinesis Data Firehose to stream data directly into S3 and use Glue to catalog it

Answer

Use a single daily Glue job to process all data at once

Question 9

A data engineer is building a streaming pipeline using Amazon Kinesis Data Streams and AWS Lambda. The Lambda function processes records and writes results to Amazon S3. The engineer notices that the Lambda function is experiencing throttling and some records are being dropped. Which TWO actions should the engineer take to improve the reliability of the pipeline?

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Option A is correct because increasing the number of shards in the Kinesis data stream raises the stream's read throughput capacity; each shard supports up to 2 MB/sec of read throughput (and 1 MB/sec or 1,000 records/sec write), and Lambda's event source mapping polls each shard with a single concurrent invocation per shard, so more shards allow more parallel Lambda invocations and reduce the chance that the stream itself becomes the bottleneck causing dropped records. Option B is correct because setting reserved concurrency on the Lambda function guarantees a dedicated concurrency pool for that function, preventing it from being throttled when other functions in the account consume the shared regional concurrency quota; this directly addresses the Lambda throttling that is causing records to be dropped. Option C is not correct here because a Dead Letter Queue captures failed invocations after retries are exhausted, but it does not prevent throttling or stop records from being dropped due to concurrency limits. Option D is not correct because decreasing the batch size reduces records per invocation but does not increase available concurrency, so throttling would persist and throughput would actually decrease. Option E is not correct because extending the Kinesis retention period to 7 days only affects how long data remains in the stream; it does not resolve Lambda throttling or prevent record drops.

Answer

Add a Dead Letter Queue to the Lambda function to capture failed records.

Answer

Decrease the batch size in the Lambda event source mapping.

Answer

Increase the Kinesis stream's retention period to 7 days.

Question 10

A machine learning team is using Amazon SageMaker to train a model on a dataset stored in S3. The training job reads data from S3 using Pipe input mode, but the training is slow. The team wants to improve data throughput. Which THREE actions should they take?

Accepted Answer

Mount the S3 bucket using an S3 file system and use File mode with a larger instance type.. Option B is correct because switching from Pipe mode to File mode with an S3 file system mount (e.g., using s3fs or Mountpoint for Amazon S3) allows the training container to access data as local files, and a larger instance type provides more memory, network bandwidth, and CPU to handle the increased I/O throughput. Option C is correct because an S3 VPC Gateway Endpoint keeps traffic between the SageMaker VPC and S3 on the AWS private network, avoiding NAT gateways and internet hops, which reduces latency and improves throughput for S3 reads. Option D is correct because Amazon EFS provides a shared, low-latency, high-throughput POSIX file system that can be mounted directly by SageMaker training jobs, eliminating S3 request overhead and enabling faster sequential and parallel reads for training data. Option A is not appropriate because S3 Transfer Acceleration speeds up uploads/downloads over the public internet from distant clients, not intra-VPC SageMaker training traffic. Option E is not appropriate because ElastiCache (Redis/Memcached) is an in-memory cache for application data, not a training data source, and would require custom integration not supported natively by SageMaker training input channels.

Answer

Enable S3 Transfer Acceleration on the bucket.

Answer

Use Amazon ElastiCache to cache the training data.

Question 11

A data engineer has attached the above IAM policy to an IAM role used by an AWS Glue ETL job. The job reads from and writes to 'my-data-bucket'. The job is failing with an Access Denied error. What is the most likely cause?

Accepted Answer

The condition restricts access to a specific IP range that does not include the AWS Glue service IPs.. The IAM policy includes a condition that restricts access to requests originating from a specific IP address range. AWS Glue ETL jobs run on ephemeral compute resources that use a dynamic pool of IP addresses, which are not guaranteed to fall within any fixed customer-managed IP range. Therefore, the condition causes the Access Denied error because the Glue service IPs are not within the allowed range.

Answer

The IAM role needs to have s3:ListBucket permission.

Answer

The IAM role does not have permission to list the bucket.

Answer

The resource ARN should include the bucket itself, not just the objects.

Question 12

A machine learning engineer is using Amazon SageMaker to train a model. The training dataset is 2 TB and is stored in Amazon S3. The engineer wants to reduce the training time by improving data loading performance. Which data ingestion mode should be used?

Accepted Answer

Pipe mode. Pipe mode is the correct choice because it streams data directly from Amazon S3 to the training container via a Unix named pipe, bypassing disk writes and reducing I/O latency. For a 2 TB dataset, this eliminates the bottleneck of downloading data to the training instance's Amazon Elastic Block Store (EBS) volume, significantly improving data loading performance and reducing overall training time.

Answer

Incremental mode

Answer

File mode

Answer

Fast file mode

Question 13

A data engineer is investigating a slow Athena query on a partitioned table. The table is partitioned by year, month, and day, and the data is stored in S3 with the prefix pattern 'raw/YYYY/MM/DD/'. The engineer runs the above CLI command and sees that there are many small files. Which action would most improve query performance?

Accepted Answer

Use S3DistCp to coalesce files into fewer, larger files.. The core issue is that the Athena query is slow due to many small files, which increases the overhead of S3 LIST operations and task scheduling in the Presto/Trino engine underlying Athena. Coalescing these small files into fewer, larger files with S3DistCp reduces the number of S3 GET requests and minimizes the scheduling overhead, directly improving query throughput. This is a classic small-files problem, not a data format or partitioning issue.

Answer

Convert the data to columnar format like Parquet or ORC.

Answer

Increase the number of partitions in the Athena DDL.

Answer

Add more partitions to reduce the amount of data scanned per query.

Question 14

A data engineering team is building a real-time fraud detection pipeline. The pipeline ingests transaction data from an Amazon Kinesis Data Stream with 10 shards. Each shard produces about 500 records per second, each record is 2 KB. The data is processed by a Lambda function that runs for about 200 ms and then writes results to an Amazon DynamoDB table. The team notices that the Lambda function is experiencing a high number of throttles, and there are increasing numbers of records being retried. The Lambda function's reserved concurrency is set to 100. The DynamoDB table has 100 read capacity units and 100 write capacity units. Which change would most effectively reduce throttling and improve processing throughput?

Accepted Answer

Increase the DynamoDB write capacity units to 1000.. The Lambda function writes to DynamoDB, and the table has only 100 write capacity units. With 10 shards producing 500 records/sec at 2 KB each, the pipeline generates roughly 5,000 records/sec, and each write consumes at least 1 WCU (more for items over 1 KB). The table's 100 WCUs are far too low, causing DynamoDB throttling, which in turn causes Lambda retries and throttles. Increasing the write capacity units to 1000 directly addresses the bottleneck.

Answer

Decrease the Lambda function's batch size to 10.

Answer

Increase the number of shards in the Kinesis stream to 100.

Answer

Increase the Lambda function's reserved concurrency to 1000.

Question 15

A data engineer is tasked with building a data pipeline that moves data from an on-premises database to Amazon S3 for analytics. The database is a MySQL instance that is 2 TB in size. The company has a 1 Gbps dedicated network connection to AWS (AWS Direct Connect). The data must be transferred once daily. The engineer needs to choose the most efficient and reliable service for this task. Which service should they use?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is the correct choice because it is purpose-built for migrating databases to AWS, supporting continuous replication and one-time migrations from MySQL to Amazon S3. It can handle the 2 TB dataset efficiently over a 1 Gbps Direct Connect link by using change data capture (CDC) for ongoing replication and parallel tasks for throughput, ensuring reliability with built-in monitoring and restart capabilities.

Answer

AWS DataSync

Answer

AWS Glue

Answer

Amazon S3 Transfer Acceleration