MLS-C01 Data Engineering — 20 Questions

Question 1

A machine learning team needs to create a training dataset by joining two large datasets (10 TB and 5 TB) stored in S3. The join key is 'user_id'. They want to minimize data movement and cost. Which approach should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use AWS Glue ETL to read both datasets, join them using Spark DataFrames, and write the result to S3.. AWS Glue ETL provides a serverless Spark environment that can read both datasets from S3, perform the join in a distributed manner, and write the result back to S3. This approach minimizes data movement because Glue reads directly from S3 and writes back to S3 without requiring intermediate storage or cluster management. It is cost-effective due to per-second billing and is optimized for large-scale ETL workloads like joining 10 TB and 5 TB datasets. Amazon Athena would scan 15 TB of data, incurring high costs and potential performance issues, while EMR and Redshift require provisioning and managing clusters, leading to higher operational overhead and costs.

Answer

Launch an Amazon EMR cluster with Spark, read data from S3, perform the join, and write results back to S3.

Answer

Use Amazon Athena to run a SQL query joining the two datasets directly on S3.

Answer

Load both datasets into Amazon Redshift using COPY commands, then perform the join in Redshift.

Question 2

A data pipeline uses Amazon Kinesis Data Streams with a Lambda consumer to process clickstream data. The Lambda function sometimes times out because of spikes in traffic. The team wants to buffer the data before processing to handle spikes. Which approach is most effective?

Accepted Answer

Send the data to an Amazon SQS queue and have Lambda poll from it. Amazon SQS can act as a buffer to handle traffic spikes. By sending data from Kinesis Data Streams to an SQS queue, Lambda can poll from the queue in batches, smoothing out the load and preventing timeouts. This decouples the ingestion from processing, allowing Lambda to process data at its own pace.

Answer

Increase the number of shards in the Kinesis stream

Answer

Use Kinesis Data Firehose as an intermediary that buffers data and delivers to Lambda in batches

Answer

Increase the Lambda function timeout to 15 minutes

Question 3

A company runs a nightly AWS Glue ETL job that processes data from an Amazon Redshift cluster and writes to Amazon S3. The job fails intermittently with 'ERROR: cannot execute INSERT in a read-only transaction'. What is the most likely cause?

Accepted Answer

The Redshift cluster is in a read-only state due to a failover or maintenance. This error occurs when the Glue job tries to write to a Redshift table that is in a read-only transaction, often because the Redshift cluster is in a read-only state due to a failover or maintenance. Option A (IAM permissions) would result in an access denied error. Option B (JDBC driver version) would typically cause a connection or compatibility error. Option D (connection pool exhaustion) would produce a different error related to connection limits.

Answer

The IAM role used by Glue does not have permissions to insert into Redshift

Answer

The JDBC driver version is incompatible with the Redshift cluster

Answer

The Glue job's connection pool is exhausted

Question 4

A data engineer is tasked with building a system to process a continuous stream of IoT sensor data. The data must be processed in near real-time, and the results must be stored in Amazon S3 partitioned by hour. Which AWS service is the most cost-effective and simplest to implement?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is the simplest and most cost-effective way to ingest streaming data and deliver it to S3 with automatic partitioning by time. Option A (Amazon SQS with AWS Lambda) is for message queuing and decoupling, not for streaming data to S3 with partitioning. Option C (Amazon Kinesis Data Streams with EC2 consumers) requires custom consumers and more management overhead. Option D (AWS DMS) is for database migration, not streaming.

Answer

Amazon Simple Queue Service (SQS) with AWS Lambda

Answer

Amazon Kinesis Data Streams with Amazon EC2 consumers

Answer

AWS Database Migration Service (DMS) for continuous replication

Question 5

A company is using AWS Glue to run ETL jobs that transform data from Amazon S3 to Amazon Redshift. The jobs are currently failing due to insufficient memory. The data volume varies, with occasional spikes. Which solution should be used to handle the variable memory requirements efficiently?

Accepted Answer

Increase the number of DPUs (Data Processing Units) for all jobs. To handle variable memory requirements efficiently, increasing the number of DPUs for all jobs ensures sufficient memory during spikes. AWS Glue Flex execution uses spare capacity and can be preempted, so it does not guarantee resource availability and is not suitable for handling memory insufficiencies. Option C provides a direct way to allocate more resources, preventing job failures due to insufficient memory.

Answer

Migrate the ETL jobs to Amazon EMR with Apache Spark

Answer

Use AWS Glue Flex execution to allocate resources dynamically

Answer

Split the jobs into smaller steps and run them sequentially

Question 6

A data pipeline uses Amazon Kinesis Data Streams to ingest event data. The data is consumed by an AWS Lambda function, which writes to Amazon DynamoDB. The Lambda function is experiencing throttling errors, and the DynamoDB write capacity is underutilized. The events must be processed in order per shard. Which solution most effectively addresses the throttling?

Accepted Answer

Increase the Lambda function's reserved concurrency. The Lambda function is throttling because the number of concurrent invocations from the Kinesis stream exceeds the function's concurrency limit (default 1000 per region). Increasing the number of shards would increase the concurrency, worsening the throttling. The correct approach is to increase the Lambda function's reserved concurrency, which raises the number of allowed concurrent executions, directly addressing the throttling while preserving per-shard ordering.

Answer

Use an SQS FIFO queue between Kinesis and Lambda to buffer

Answer

Increase the write capacity units of the DynamoDB table

Answer

Increase the number of shards in the Kinesis Data Stream

Question 7

An ML engineer runs the AWS CLI command shown in the exhibit on a file in S3. The engineer wants to use this file in a SageMaker training job. What does the output reveal about the data?

Accepted Answer

The file is 5 GB and is stored as a CSV. The AWS CLI command `aws s3 ls s3://bucket/path/file.csv --summarize --human-readable` lists the object's size in human-readable format (e.g., 5 GB) and its last modified date. The output does not include ContentLength or version ID; however, the file extension `.csv` indicates the file is a CSV. This size and format information is sufficient to confirm the file is a 5 GB CSV, which can be used directly in a SageMaker training job. Options B (5 MB) is incorrect because the output shows 5 GB. Option C (Parquet) is incorrect because the extension is `.csv`. Option D (versioned) is incorrect because `aws s3 ls` without `--versions` does not show version ID.

Answer

The file is 5 MB and is stored as a CSV

Answer

The file is in Parquet format with two features and one label

Answer

The file is versioned and can be accessed by version ID

Question 8

A company runs a data lake on Amazon S3 with partitions by year/month/day. A machine learning team needs to read daily data from the last 30 days for model retraining. The data format is Parquet. The team uses Amazon Athena to query the data, but the queries are slow and scanning too much data. The team has already optimized the file sizes and compression. What additional step can reduce the amount of data scanned?

Accepted Answer

Add more partition columns such as hour to reduce the scanned partitions.. Adding more granular partition columns, such as hour, enables more precise partition pruning in Athena. If the query includes a filter on the hour column (e.g., WHERE hour BETWEEN 9 AND 17), Athena will only read the partitions corresponding to those hours, reducing the amount of data scanned. Even if the current query does not filter on hour, this additional partition provides flexibility to reduce scanned data when only specific hours are needed per day. Since the team can modify their query to include an hour filter, this step can significantly reduce the data scanned, especially when combined with existing year/month/day partitions.

Answer

Remove the partition structure and store data as single large files.

Answer

Convert the Parquet files to JSON format for better query performance.

Answer

Use CSV format with Gzip compression.

Question 9

A company is using AWS Glue to run ETL jobs that process data from an Amazon RDS for PostgreSQL database. The jobs are failing with connection timeouts. The security group for the RDS instance allows inbound traffic from the Glue job's security group. What is the most likely cause?

Accepted Answer

The VPC does not have an S3 VPC endpoint. The most likely cause is that the VPC does not have an S3 VPC endpoint. AWS Glue jobs running in a VPC require access to S3 for storing temporary data, scripts, and logs. Without an S3 VPC endpoint, traffic to S3 is routed through a public IP, which fails in private subnets lacking internet connectivity. This failure can cause the Glue job to hang or fail during initialization, and the attempted connection to RDS may time out as a result. The security group inbound rule is correctly configured, so ingress is not the issue. The problem is network egress for S3, not access to RDS itself.

Answer

The security group inbound rule is incorrect

Answer

RDS is in a different subnet

Answer

The IAM role does not have permission to access RDS

Question 10

A data engineering team is building a real-time fraud detection pipeline. The pipeline ingests transaction data from an Amazon Kinesis Data Stream with 10 shards. Each shard produces about 500 records per second, each record is 2 KB. The data is processed by a Lambda function that runs for about 200 ms and then writes results to an Amazon DynamoDB table. The team notices that the Lambda function is experiencing a high number of throttles, and there are increasing numbers of records being retried. The Lambda function's reserved concurrency is set to 100. The DynamoDB table has 100 read capacity units and 100 write capacity units. Which change would most effectively reduce throttling and improve processing throughput?

Accepted Answer

Increase the DynamoDB write capacity units to 1000.. The observed throttling is due to insufficient Lambda concurrency. With 10 shards each producing 500 records per second and a batch size of 100, each shard generates 5 batches per second. Given a 200 ms processing time, each shard requires multiple concurrent invocations to avoid backlog, but the current reserved concurrency of 100 may limit the number of simultaneous executions. Increasing reserved concurrency to 1000 ensures the function can process all batches in parallel without throttling (e.g., via ParallelizationFactor >1 if configured), reducing retries and improving throughput. DynamoDB write capacity may also be a factor, but the most direct remedy for Lambda throttling is raising the concurrency limit.

Answer

Decrease the Lambda function's batch size to 10.

Answer

Increase the number of shards in the Kinesis stream to 100.

Answer

Increase the Lambda function's reserved concurrency to 1000.

Question 11

Refer to the exhibit. A team deploys this CloudFormation stack. The Kinesis stream is created, but the Firehose delivery stream fails to create with a 'Resource handler returned message: Unable to assume role' error. What is the most likely cause?

Accepted Answer

The IAM role 'firehose-role' does not exist in the AWS account.. The Firehose role 'firehose-role' is specified with a static ARN that includes an account ID. If the stack is deployed in a different account, or if the role does not exist, the assumption fails. Option D (role does not exist in the account) is the most likely. Option A (shard count) is not related. Option B (encryption) is not related. Option C (retention period) is fine.

Answer

The shard count of 2 is too low for Firehose to read from.

Answer

The Kinesis stream is encrypted with KMS, but Firehose does not have permission to decrypt.

Answer

The retention period of 168 hours is too long for Firehose.

Question 12

A data engineer is designing a data lake on Amazon S3. The data comes from various sources and must be stored in a way that supports both batch and real-time analytics. The engineer needs to partition the data to optimize query performance in Amazon Athena. Which partitioning strategy is MOST appropriate?

Accepted Answer

Partition by year, then month, then day, then hour. Partitioning by year, month, day, and hour allows efficient query pruning for both batch (e.g., daily aggregations) and real-time (e.g., hourly) access patterns. Option A (hash of record ID) creates random partitions and is not beneficial for time-range queries. Option B (single prefix) results in full table scans and poor performance. Option D (source type then date) might cause partition skew and is less optimal for time-series analysis compared to a pure time-based hierarchy.

Answer

Partition by a hash of the record ID to distribute data evenly

Answer

Do not partition; use a single prefix for all data

Answer

Partition by source type and then by date

Question 13

A company stores sensitive customer data in Amazon S3. The company must ensure that data is encrypted at rest. The company also needs to manage the encryption keys using an AWS service that allows automatic rotation of keys. Which solution meets these requirements?

Accepted Answer

Use Server-Side Encryption with AWS KMS-Managed Keys (SSE-KMS) and enable automatic key rotation. AWS KMS with automatic key rotation meets the encryption and key management requirements. Option A is wrong because client-side encryption with AWS CloudHSM requires managing keys on the client side, and CloudHSM does not provide automatic key rotation. Option C is wrong because SSE-C requires the customer to provide and manage their own keys, and automatic rotation is not supported. Option D is wrong because SSE-S3 uses Amazon S3-managed keys, which cannot be managed by the customer and do not provide automatic key rotation control.

Answer

Use client-side encryption with AWS CloudHSM

Answer

Use Server-Side Encryption with Customer-Provided Keys (SSE-C)

Answer

Use Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)

Question 14

A data engineering team uses Amazon EMR with Spark to transform large datasets in S3. The team notices that the Spark jobs on the EMR cluster are failing with out-of-memory errors. The cluster uses instance types with moderate memory. Which configuration change would MOST effectively reduce memory pressure without increasing cost?

Accepted Answer

Enable shuffle compression and set spark.shuffle.compress to true.. Enabling shuffle compression (spark.shuffle.compress=true) reduces the amount of data written to disk and stored in memory during shuffle operations, thereby lowering memory pressure without changing the cluster size or cost. Option A is incorrect because adding more instances increases cost and does not directly reduce memory per task. Option B is incorrect because increasing executor cores may increase parallelism but can also increase memory usage per task if not properly configured, and does not directly reduce memory pressure. Option C is incorrect because while Kryo serialization reduces object serialization overhead, it does not significantly reduce shuffle data size; shuffle compression is more effective for reducing memory usage during shuffle.

Answer

Use a larger EMR cluster with more instances of the same type.

Answer

Increase the number of executor cores to improve parallelism.

Answer

Switch from Java serialization to Kryo serialization.

Question 15

A company runs a nightly batch job that reads data from Amazon RDS for PostgreSQL, transforms it using AWS Glue, and writes the output to Amazon S3 in Parquet format. The job takes 2 hours to complete, but the data volume has grown, and the job now takes 4 hours, exceeding the allowed window. The team needs to reduce the job duration without increasing cost. Which action is MOST effective?

Accepted Answer

Partition the output S3 data by a high-cardinality column used in filtering during transformation.. Increasing DPUs would increase cost, which violates the requirement to not increase cost. Option B is incorrect because job bookmarks are designed for incremental processing, but the job runs a full batch load nightly, so bookmarks do not skip data in a full load. Option C is incorrect because converting from Parquet to CSV increases file size and write time, likely lengthening the job. Option D is correct: partitioning the output by a high-cardinality column used in filtering during transformation can reduce write time by enabling parallel writes and can improve downstream query performance, but more importantly it may allow the transformation to leverage partition pruning if the job reads the output back or if the transformation logic can use the partition column to reduce data shuffling. However, the primary benefit is that it optimizes storage and retrieval without additional cost, making it the best choice among the options.

Answer

Increase the number of Glue DPUs (Data Processing Units) allocated to the job.

Answer

Enable AWS Glue job bookmark to skip already processed data.

Answer

Change the output format from Parquet to CSV to reduce write time.

Question 16

A company uses Amazon Athena to analyze data stored in S3. The data is in CSV format and is partitioned by year/month/day. Queries that filter on a specific day are slow. The team wants to improve query performance without changing the data format. Which action should the team take?

Accepted Answer

Ensure that the S3 folder structure follows the Hive-style partition naming convention (e.g., year=2023/month=01/day=01).. Partitioning pruning is most effective when the partition columns are in a Hive-style format (e.g., year=2023/month=01/day=01). If the folders are not in that format, Athena may not prune partitions properly. Option A is wrong because increasing the number of partitions would further slow queries. Option B is wrong because converting to Parquet changes the format. Option C is wrong because using a large number of small files hurts performance.

Answer

Use a larger number of small files to increase parallelism.

Answer

Increase the number of partitions by adding hour and minute levels.

Answer

Convert the data to Parquet format.

Question 17

A company wants to build a data lake on Amazon S3. The data lake will store raw data in its original format and also transformed data in Parquet. The data is generated by various sources and must be cataloged for discovery. Which service should the company use to automatically discover, catalog, and make the data searchable?

Accepted Answer

AWS Glue Data Catalog. AWS Glue Data Catalog is a central metadata repository that can automatically crawl S3 data sources to discover, catalog, and make data searchable. It is the correct choice for building a data lake with raw and transformed data in Parquet. Option B (Amazon S3) is storage, not a catalog. Option C (Amazon Athena) is a query engine, not a catalog. Option D (Amazon EMR) is a big data processing framework, not a catalog.

Answer

Amazon S3

Answer

Amazon Athena

Answer

Amazon EMR

Question 18

A data engineer is tasked with building a pipeline to process streaming data from IoT devices. The devices send data in JSON format every second. The pipeline must aggregate data in 5-minute windows and store the results in Amazon S3. The engineer needs to handle late-arriving data (up to 1 hour) and ensure exactly-once semantics. Which combination of AWS services should they use?

Accepted Answer

Amazon Kinesis Data Streams for ingestion, Amazon Kinesis Data Analytics for windowed aggregations, and Amazon Kinesis Data Firehose to write to Amazon S3.. Amazon Kinesis Data Analytics supports windowed aggregations via SQL or Apache Flink, can handle late-arriving data using watermarking, and when used with Kinesis Data Streams as the source, provides exactly-once processing semantics. Option B is incorrect because AWS Glue Streaming ETL is based on Spark Streaming; while it can perform windowed aggregations, it does not natively guarantee exactly-once semantics with Kinesis Data Streams and is more complex to configure for late data handling. Option C is incorrect because Amazon SQS is not designed for high-throughput streaming ingestion (IoT devices sending data every second), and AWS Lambda invocations are at-least-once, not exactly-once, making it unsuitable for the required exactly-once semantics. Option D is incorrect because Amazon Kinesis Data Firehose is a delivery service that does not support custom windowed aggregations; it delivers data after a minimum buffer interval of 60 seconds and cannot handle late-arriving data with exactly-once guarantees.

Answer

Amazon Kinesis Data Streams for ingestion, AWS Glue Streaming ETL for aggregation, and Amazon S3 for storage.

Answer

Amazon SQS for ingestion, AWS Lambda for aggregation, and Amazon S3 for storage.

Answer

Amazon Kinesis Data Streams for ingestion, Amazon Kinesis Data Firehose for transformation, and Amazon S3 for storage.

Question 19

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a 100 Mbps internet connection and the data must be transferred within 5 days. Which AWS service is best suited for this task?

Accepted Answer

AWS Snowball Edge. AWS Snowball Edge is a physical device designed for large-scale data transfer, ideal for moving 50 TB of data when network bandwidth (100 Mbps) is insufficient to meet the 5-day deadline. Option A is wrong because AWS DataSync is a network-based service; transferring 50 TB over 100 Mbps would take approximately 50 days, far exceeding 5 days. Option C is wrong because Amazon S3 Transfer Acceleration improves transfer speeds over the internet but cannot overcome the 100 Mbps bandwidth limitation; it would still take too long. Option D is wrong because AWS Storage Gateway is for hybrid cloud storage with a local cache, not for one-time bulk data migration.

Answer

AWS DataSync

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Storage Gateway

Question 20

A company is migrating on-premises data to AWS. The data includes both structured and unstructured files, totaling 200 TB. The company has a 1 Gbps dedicated network connection to AWS. They want to minimize migration time and cost. Which TWO AWS services or features should they use together? (Choose two.)

Accepted Answer

AWS Snowball Edge. Correct options: A and D. AWS DataSync can efficiently transfer data over the network to S3, and Snowball Edge can be used for the largest files to reduce network load. Option B (S3 Transfer Acceleration) is for speeding up internet transfers, not needed over a dedicated connection. Option C (AWS Glue) is used for ETL, not for bulk data transfer. Option E (AWS Direct Connect) is already implied by the 1 Gbps dedicated connection; it's not an additional service.

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Glue

Answer

AWS Direct Connect