MLS-C01 Data Engineering — Timed 10 Questions

Question 1

A machine learning team needs to create a training dataset by joining two large datasets (10 TB and 5 TB) stored in S3. The join key is 'user_id'. They want to minimize data movement and cost. Which approach should they use?

10 minute time limit — choose an answer to begin.

Accepted Answer

Use AWS Glue ETL to read both datasets, join them using Spark DataFrames, and write the result to S3.. AWS Glue ETL provides a serverless Spark environment that can read both datasets from S3, perform the join in a distributed manner, and write the result back to S3. This approach minimizes data movement because Glue reads directly from S3 and writes back to S3 without requiring intermediate storage or cluster management. It is cost-effective due to per-second billing and is optimized for large-scale ETL workloads like joining 10 TB and 5 TB datasets. Amazon Athena would scan 15 TB of data, incurring high costs and potential performance issues, while EMR and Redshift require provisioning and managing clusters, leading to higher operational overhead and costs.

Answer

Launch an Amazon EMR cluster with Spark, read data from S3, perform the join, and write results back to S3.

Answer

Use Amazon Athena to run a SQL query joining the two datasets directly on S3.

Answer

Load both datasets into Amazon Redshift using COPY commands, then perform the join in Redshift.

Question 2

A data pipeline uses Amazon Kinesis Data Streams with a Lambda consumer to process clickstream data. The Lambda function sometimes times out because of spikes in traffic. The team wants to buffer the data before processing to handle spikes. Which approach is most effective?

Accepted Answer

Send the data to an Amazon SQS queue and have Lambda poll from it. Amazon SQS can act as a buffer to handle traffic spikes. By sending data from Kinesis Data Streams to an SQS queue, Lambda can poll from the queue in batches, smoothing out the load and preventing timeouts. This decouples the ingestion from processing, allowing Lambda to process data at its own pace.

Answer

Increase the number of shards in the Kinesis stream

Answer

Use Kinesis Data Firehose as an intermediary that buffers data and delivers to Lambda in batches

Answer

Increase the Lambda function timeout to 15 minutes

Question 3

A company runs a nightly AWS Glue ETL job that processes data from an Amazon Redshift cluster and writes to Amazon S3. The job fails intermittently with 'ERROR: cannot execute INSERT in a read-only transaction'. What is the most likely cause?

Accepted Answer

The Redshift cluster is in a read-only state due to a failover or maintenance. This error occurs when the Glue job tries to write to a Redshift table that is in a read-only transaction, often because the Redshift cluster is in a read-only state due to a failover or maintenance. Option A (IAM permissions) would result in an access denied error. Option B (JDBC driver version) would typically cause a connection or compatibility error. Option D (connection pool exhaustion) would produce a different error related to connection limits.

Answer

The IAM role used by Glue does not have permissions to insert into Redshift

Answer

The JDBC driver version is incompatible with the Redshift cluster

Answer

The Glue job's connection pool is exhausted

Question 4

A data engineer is tasked with building a system to process a continuous stream of IoT sensor data. The data must be processed in near real-time, and the results must be stored in Amazon S3 partitioned by hour. Which AWS service is the most cost-effective and simplest to implement?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is the simplest and most cost-effective way to ingest streaming data and deliver it to S3 with automatic partitioning by time. Option A (Amazon SQS with AWS Lambda) is for message queuing and decoupling, not for streaming data to S3 with partitioning. Option C (Amazon Kinesis Data Streams with EC2 consumers) requires custom consumers and more management overhead. Option D (AWS DMS) is for database migration, not streaming.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda

Answer

Amazon Kinesis Data Streams with Amazon EC2 consumers

Answer

AWS Database Migration Service (DMS) for continuous replication

Question 5

A company is using AWS Glue to run ETL jobs that transform data from Amazon S3 to Amazon Redshift. The jobs are currently failing due to insufficient memory. The data volume varies, with occasional spikes. Which solution should be used to handle the variable memory requirements efficiently?

Accepted Answer

Increase the number of DPUs (Data Processing Units) for all jobs. To handle variable memory requirements efficiently, increasing the number of DPUs for all jobs ensures sufficient memory during spikes. AWS Glue Flex execution uses spare capacity and can be preempted, so it does not guarantee resource availability and is not suitable for handling memory insufficiencies. Option C provides a direct way to allocate more resources, preventing job failures due to insufficient memory.

Answer

Migrate the ETL jobs to Amazon EMR with Apache Spark

Answer

Use AWS Glue Flex execution to allocate resources dynamically

Answer

Split the jobs into smaller steps and run them sequentially

Question 6

A data pipeline uses Amazon Kinesis Data Streams to ingest event data. The data is consumed by an AWS Lambda function, which writes to Amazon DynamoDB. The Lambda function is experiencing throttling errors, and the DynamoDB write capacity is underutilized. The events must be processed in order per shard. Which solution most effectively addresses the throttling?

Accepted Answer

Increase the Lambda function's reserved concurrency. The Lambda function is throttling because the number of concurrent invocations from the Kinesis stream exceeds the function's concurrency limit (default 1000 per region). Increasing the number of shards would increase the concurrency, worsening the throttling. The correct approach is to increase the Lambda function's reserved concurrency, which raises the number of allowed concurrent executions, directly addressing the throttling while preserving per-shard ordering.

Answer

Use an SQS FIFO queue between Kinesis and Lambda to buffer

Answer

Increase the write capacity units of the DynamoDB table

Answer

Increase the number of shards in the Kinesis Data Stream

Question 7

An ML engineer runs the AWS CLI command shown in the exhibit on a file in S3. The engineer wants to use this file in a SageMaker training job. What does the output reveal about the data?

Accepted Answer

The file is 5 GB and is stored as a CSV. The AWS CLI command `aws s3 ls s3://bucket/path/file.csv --summarize --human-readable` lists the object's size in human-readable format (e.g., 5 GB) and its last modified date. The output does not include ContentLength or version ID; however, the file extension `.csv` indicates the file is a CSV. This size and format information is sufficient to confirm the file is a 5 GB CSV, which can be used directly in a SageMaker training job. Options B (5 MB) is incorrect because the output shows 5 GB. Option C (Parquet) is incorrect because the extension is `.csv`. Option D (versioned) is incorrect because `aws s3 ls` without `--versions` does not show version ID.

Answer

The file is 5 MB and is stored as a CSV

Answer

The file is in Parquet format with two features and one label

Answer

The file is versioned and can be accessed by version ID

Question 8

A company runs a data lake on Amazon S3 with partitions by year/month/day. A machine learning team needs to read daily data from the last 30 days for model retraining. The data format is Parquet. The team uses Amazon Athena to query the data, but the queries are slow and scanning too much data. The team has already optimized the file sizes and compression. What additional step can reduce the amount of data scanned?

Accepted Answer

Add more partition columns such as hour to reduce the scanned partitions.. Adding more granular partition columns, such as hour, enables more precise partition pruning in Athena. If the query includes a filter on the hour column (e.g., WHERE hour BETWEEN 9 AND 17), Athena will only read the partitions corresponding to those hours, reducing the amount of data scanned. Even if the current query does not filter on hour, this additional partition provides flexibility to reduce scanned data when only specific hours are needed per day. Since the team can modify their query to include an hour filter, this step can significantly reduce the data scanned, especially when combined with existing year/month/day partitions.

Answer

Remove the partition structure and store data as single large files.

Answer

Convert the Parquet files to JSON format for better query performance.

Answer

Use CSV format with Gzip compression.

Question 9

A company is using AWS Glue to run ETL jobs that process data from an Amazon RDS for PostgreSQL database. The jobs are failing with connection timeouts. The security group for the RDS instance allows inbound traffic from the Glue job's security group. What is the most likely cause?

Accepted Answer

The VPC does not have an S3 VPC endpoint. The most likely cause is that the VPC does not have an S3 VPC endpoint. AWS Glue jobs running in a VPC require access to S3 for storing temporary data, scripts, and logs. Without an S3 VPC endpoint, traffic to S3 is routed through a public IP, which fails in private subnets lacking internet connectivity. This failure can cause the Glue job to hang or fail during initialization, and the attempted connection to RDS may time out as a result. The security group inbound rule is correctly configured, so ingress is not the issue. The problem is network egress for S3, not access to RDS itself.

Answer

The security group inbound rule is incorrect

Answer

RDS is in a different subnet

Answer

The IAM role does not have permission to access RDS

Question 10

A data engineering team is building a real-time fraud detection pipeline. The pipeline ingests transaction data from an Amazon Kinesis Data Stream with 10 shards. Each shard produces about 500 records per second, each record is 2 KB. The data is processed by a Lambda function that runs for about 200 ms and then writes results to an Amazon DynamoDB table. The team notices that the Lambda function is experiencing a high number of throttles, and there are increasing numbers of records being retried. The Lambda function's reserved concurrency is set to 100. The DynamoDB table has 100 read capacity units and 100 write capacity units. Which change would most effectively reduce throttling and improve processing throughput?

Accepted Answer

Increase the DynamoDB write capacity units to 1000.. The observed throttling is due to insufficient Lambda concurrency. With 10 shards each producing 500 records per second and a batch size of 100, each shard generates 5 batches per second. Given a 200 ms processing time, each shard requires multiple concurrent invocations to avoid backlog, but the current reserved concurrency of 100 may limit the number of simultaneous executions. Increasing reserved concurrency to 1000 ensures the function can process all batches in parallel without throttling (e.g., via ParallelizationFactor >1 if configured), reducing retries and improving throughput. DynamoDB write capacity may also be a factor, but the most direct remedy for Lambda throttling is raising the concurrency limit.

Answer

Decrease the Lambda function's batch size to 10.

Answer

Increase the number of shards in the Kinesis stream to 100.

Answer

Increase the Lambda function's reserved concurrency to 1000.