MLS-C01 Data Engineering — 20 Questions

Question 1

A company uses Amazon Kinesis Data Streams to ingest real-time clickstream data from a website. The data is consumed by a Lambda function that writes records to an S3 bucket. Recently, the number of shards was increased from 2 to 4 to handle higher throughput. After the change, the Lambda function started processing records with increased latency and some records were being written out of order. What is the MOST likely cause?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The partition key used by the producer does not ensure that related records go to the same shard after resharding.. After resharding from 2 to 4 shards, the mapping of partition keys to shards changes. If the producer does not use a partition key that ensures related records (e.g., same user session) are routed to the same shard, records that were previously ordered within a shard may now be split across multiple shards. Since the Lambda consumer processes shards independently, records from the same logical sequence can arrive out of order, and the increased shard count can also cause higher latency if the consumer is not properly parallelized.

Answer

The S3 bucket is not configured with versioning, causing overwrites.

Answer

The Lambda function is reading from the oldest sequence number, causing high IteratorAgeSeconds.

Answer

The Lambda function’s reserved concurrency is too low for the increased shard count.

Question 2

An ML team is building a recommendation system. The training data includes user-item interactions stored in Amazon DynamoDB. The team wants to export this data to S3 in Parquet format for use with Amazon SageMaker. The export should be incremental (only new or changed records) and run daily. Which approach meets these requirements with MINIMAL operational overhead?

Accepted Answer

Use DynamoDB Streams with AWS Lambda to write changes to S3 in Parquet format.. DynamoDB Streams capture every change (insert, update, delete) in near real-time, and AWS Lambda can process these events to write only the changed records to S3 in Parquet format. This approach provides incremental, daily exports with minimal operational overhead, as it is fully serverless and requires no infrastructure management.

Answer

Use the DynamoDB Export to S3 feature and schedule it daily with AWS Glue.

Answer

Use a script that scans the DynamoDB table and filters by last updated timestamp.

Answer

Set up an Amazon EMR cluster running Spark jobs to read DynamoDB and write to S3.

Question 3

A company uses AWS Glue ETL jobs to process data from multiple sources. The job fails with the error: 'An error occurred while calling o123.pyWriteDynamicFrame. Insufficient memory.' The job runs on a G.1X worker type with 10 workers. What should be changed to resolve this error?

Accepted Answer

Increase the number of workers to 20.. The error 'Insufficient memory' in AWS Glue ETL jobs typically indicates that the total memory across all executors is insufficient for the data being processed. Increasing the number of workers from 10 to 20 doubles the total memory and compute capacity available, allowing the job to handle larger datasets without running out of memory. This is the most direct and effective fix for a memory exhaustion error when using the G.1X worker type.

Answer

Enable the Spark UI to monitor the job.

Answer

Change the worker type to G.2X.

Answer

Reduce the number of partitions in the DynamicFrame.

Question 4

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to a sink. The application is failing with an 'OutOfMemoryError'. The application has parallelism set to 4 and uses 1 Kinesis Processing Unit (KPU). What is the MOST likely cause and solution?

Accepted Answer

The heap memory per operator is too low; increase parallelism to 8.. With parallelism set to 4 but only 1 KPU, each operator slot receives a fraction of the available heap memory, leading to an OutOfMemoryError. Increasing parallelism to 8 distributes the workload across more slots, but more importantly, it forces Kinesis Data Analytics to allocate additional KPUs (each KPU provides 4 GB of memory), thereby increasing the total heap memory available to the application.

Answer

The application is using too many operators; reduce parallelism to 2.

Answer

The checkpoint interval is too short; increase it to 5 minutes.

Answer

The buffer timeout is too high; reduce it to 50 ms.

Question 5

A data science team is building a real-time fraud detection system. Transactions are streamed via Amazon Kinesis Data Streams, and a Lambda function performs feature engineering and invokes an Amazon SageMaker endpoint for predictions. The team notices that the Lambda function is timing out and causing data loss. Which solution should the team implement to process the stream reliably and at low latency?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to consume the stream, perform feature engineering, and invoke the SageMaker endpoint with exactly-once processing.. Amazon Kinesis Data Analytics for Apache Flink provides a stateful, low-latency stream processing engine that can consume from Kinesis Data Streams, perform feature engineering in real-time, and invoke SageMaker endpoints with exactly-once processing semantics. This eliminates Lambda timeouts and data loss by using a long-running, scalable application instead of a short-lived function.

Answer

Use the Kinesis Client Library (KCL) to process the stream in an Amazon EC2 instance, and store the predictions in Amazon DynamoDB.

Answer

Increase the Lambda function timeout to 15 minutes and allocate more memory to reduce processing time.

Answer

Configure Amazon Kinesis Firehose to deliver the stream to an Amazon S3 bucket, then trigger a Lambda function to process the data in batches.

Question 6

A data engineer is building a data pipeline to process user clickstream data. The data arrives as JSON files in an S3 bucket. The pipeline must transform the JSON into Parquet format and partition by date and event type, then make the data available for Amazon Athena queries. The engineer needs a fully managed, serverless solution with minimal operational overhead. Which combination of AWS services should the engineer use?

Accepted Answer

Use S3 Event Notifications to trigger an AWS Lambda function that converts the JSON to Parquet and writes to a partitioned S3 location, then query with Athena.. AWS Lambda triggered by S3 Event Notifications provides a fully serverless, event-driven architecture with minimal operational overhead for converting JSON to Parquet and partitioning by date and event type. Lambda can process each new JSON file as it arrives, perform the transformation in memory (using libraries like PyArrow or Pandas), and write the Parquet output to a partitioned S3 path, which Athena can then query directly. This approach avoids managing any clusters or job scheduling, aligning with the requirement for a fully managed, serverless solution.

Answer

Use Amazon EMR with Spark to read JSON, convert to Parquet, and partition, then query with Athena.

Answer

Use AWS Glue ETL jobs to read JSON from S3, transform to Parquet, and write to a partitioned S3 location, then use Athena.

Answer

Use Amazon Kinesis Firehose to ingest data and convert to Parquet, then write to S3, and query with Athena.

Question 7

A team is using Amazon SageMaker to train a model on a dataset that is 500 GB in size, stored as CSV files in S3. The training job takes 2 hours using a single ml.p3.2xlarge instance. The team wants to reduce training time to under 30 minutes. The model architecture supports distributed training. Which solution will achieve this goal with the LEAST amount of code changes?

Accepted Answer

Use multiple ml.p3.2xlarge instances with SageMaker's distributed data parallelism library, enabling automatic sharding of the training data.. SageMaker's distributed data parallelism library automatically shards the training data across multiple ml.p3.2xlarge instances, enabling parallel gradient computation and reducing wall-clock training time from 2 hours to under 30 minutes without requiring manual code changes to the training script. The model architecture already supports distributed training, so the library handles the communication and synchronization (e.g., AllReduce) transparently.

Answer

Use managed spot training to reduce cost and then use cost savings to train with a larger instance.

Answer

Use a single ml.p3.16xlarge instance with more GPUs and memory.

Answer

Change the input mode to Pipe mode to stream data from S3 directly, reducing I/O wait time.

Question 8

A company processes large streams of IoT sensor data using Amazon Kinesis Data Streams with 100 shards. Each sensor reading is about 1 KB. The data is consumed by an Amazon EMR cluster running Spark Streaming jobs. The team notices that the Spark Streaming job's processing time is gradually increasing, and the stream is falling behind. They suspect the issue is due to skewed data distribution across shards. Which approach should the team take to diagnose and resolve the issue?

Accepted Answer

Modify the producer to add a random prefix to the partition key, ensuring even distribution across all shards, and monitor the stream using CloudWatch.. Adding a random prefix to the partition key ensures that sensor data is evenly distributed across all 100 shards, eliminating hot shards that cause processing delays. This directly addresses the skewed data distribution issue without requiring infrastructure changes, and the team can monitor the improvement using CloudWatch metrics like IncomingBytes and ReadProvisionedThroughputExceeded.

Answer

Increase the number of shards to 200 to provide more parallelism.

Answer

Check Amazon CloudWatch metrics for Kinesis to identify hot shards, then manually redistribute the data by repartitioning in Spark.

Answer

Use the Kinesis Client Library (KCL) with a custom worker to rebalance the load across shards.

Question 9

A company is building a real-time anomaly detection system for network traffic logs. The logs are ingested via Amazon Kinesis Data Streams and processed with an Amazon SageMaker endpoint for inference. The team needs to ensure that the inference results are stored durably and can be replayed for model retraining. The system must handle at least 10,000 records per second with low latency. Which three AWS services should the team use to build this architecture? (Select THREE.)

Accepted Answer

Amazon Kinesis Data Streams. Option B (Amazon Kinesis Data Streams) is correct because it is the ingestion backbone that captures the network traffic logs at high throughput (10,000+ records/sec) with low latency and retains records for replay. Option C (Amazon Kinesis Data Analytics for Apache Flink) is correct because it provides real-time stream processing to run anomaly detection logic over the incoming records before or alongside inference, and it integrates natively with Kinesis Data Streams. Option E (Amazon SageMaker) is correct because it hosts the inference endpoint that scores the processed records, and its outputs can be written durably to a store such as Amazon S3 for replay and retraining. Option A (AWS Glue ETL) is not appropriate here because it is a batch/serverless ETL service rather than a low-latency streaming processor. Option D (Amazon Kinesis Data Firehose) is not selected because, while it can deliver data to destinations like S3, it is a managed delivery service rather than the stream-processing or inference component needed for real-time anomaly detection.

Answer

AWS Glue ETL

Answer

Amazon Kinesis Data Firehose

Question 10

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Lambda function that writes to an S3 bucket. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors. What is the MOST likely cause?

Accepted Answer

The Kinesis stream has too few shards for the data volume.. The 'ProvisionedThroughputExceededException' error in Amazon Kinesis Data Streams indicates that the data ingestion rate exceeds the write capacity of the stream's shards. Each shard supports up to 1 MB/s or 1,000 records/s for writes. If the clickstream data volume surpasses this limit, the Lambda function, which reads from the stream, will encounter this exception. Increasing the number of shards scales the write capacity to match the data volume.

Answer

The data retention period of the stream is too short.

Answer

The S3 bucket has insufficient write capacity.

Answer

The Lambda function's reserved concurrency is set too high.

Question 11

A company uses AWS Glue to catalog data in S3. Data is partitioned by year, month, day. The Glue crawler runs daily but sometimes misses new partitions. What should be done to ensure all partitions are cataloged?

Accepted Answer

Configure the crawler to update all partitions on each run.. Configuring the Glue crawler to 'update all partitions on each run' forces the crawler to scan the entire S3 path and register any new partitions it finds, even if the partition structure hasn't changed. This ensures all missed partitions are cataloged. Partition indexing (D) improves query performance by creating an index over existing partitions, but does not automatically discover new partitions.

Answer

Use a custom classifier to detect partition patterns.

Answer

Increase the crawler schedule to run every hour.

Answer

Enable partition indexing in the Glue table properties.

Question 12

A data engineer created an IAM policy to allow a Glue ETL job to read and write objects to an S3 bucket. The ETL job fails when writing data with the error 'Access Denied'. The job is configured to use SSE-S3 (AES256) encryption. What is the likely issue?

Accepted Answer

The policy does not grant s3:PutObject on the bucket itself, which is needed for some write operations.. The error 'Access Denied' when writing to S3 with SSE-S3 encryption typically occurs because the IAM policy lacks the `s3:PutObject` permission on the bucket resource itself. While the policy may grant `s3:PutObject` on the object ARN (`arn:aws:s3:::bucket/*`), some S3 write operations—especially those involving encryption headers or bucket-level checks—also require the permission on the bucket ARN (`arn:aws:s3:::bucket`). Without this, the request is denied even if the object-level permission exists.

Answer

The policy grants s3:PutObject on all buckets, not just the specific one.

Answer

The condition requires objects to be encrypted with SSE-KMS, but the job uses SSE-S3.

Answer

The condition requires objects to use SSE-S3, but the job uses SSE-KMS.

Question 13

A company runs a real-time fraud detection system using Amazon Kinesis Data Streams with 100 shards. Data is consumed by a custom Java application running on Amazon EC2 instances in an Auto Scaling group. The application processes records and writes results to a DynamoDB table. Over the past month, the application has experienced intermittent slowdowns and the DynamoDB write capacity has been fully utilized during peak hours. The team wants to improve throughput without losing the ability to reprocess failed records. The application currently uses the Kinesis Client Library (KCL) with DynamoDB as the lease table. The team is considering the following changes: A. Increase the number of EC2 instances to match the number of shards. B. Switch to using AWS Lambda as the consumer to handle scaling automatically. C. Increase the write capacity of the DynamoDB lease table to handle more workers. D. Use enhanced fan-out to have each consumer receive its own 2 MB/second shard throughput. Which change should the team implement first to address the issue?

Accepted Answer

Use enhanced fan-out to have each consumer receive its own 2 MB/second shard throughput.. The primary bottleneck is DynamoDB write capacity being fully utilized during peak hours. Enhanced fan-out (option B) provides each consumer with a dedicated 2 MB/second read throughput per shard, eliminating the need for consumers to contend for the shared 2 MB/second per shard. This reduces the load on the DynamoDB lease table because workers no longer need to poll for records, which in turn lowers the write operations to the lease table and alleviates the DynamoDB write capacity issue.

Answer

Increase the write capacity of the DynamoDB lease table to handle more workers.

Answer

Switch to using AWS Lambda as the consumer to handle scaling automatically.

Answer

Increase the number of EC2 instances to match the number of shards.

Question 14

A retail company runs an e-commerce platform on AWS. They have a Data Engineering team that processes clickstream data using Amazon Kinesis Data Streams (KDS) with a shard count of 5. The data is consumed by an AWS Lambda function that transforms and loads the data into an Amazon S3 bucket partitioned by year/month/day/hour. Recently, the team has noticed that the Lambda function is experiencing throttling errors, and the KDS shard iterator age is increasing, indicating that the consumer cannot keep up with the incoming data rate. The team has already increased the Lambda reserved concurrency to 1000 and enabled batch window of 60 seconds. The metrics show that the Lambda function duration is well under the 5-minute timeout, and there are no errors in the transformation logic. The S3 write operations are not failing. Which course of action would MOST effectively resolve the issue without unnecessary cost or complexity?

Accepted Answer

Increase the number of shards in the Kinesis Data Stream to 20 to increase the parallelism of Lambda consumers.. The core issue is that the Lambda consumer cannot keep up with the incoming data rate, as evidenced by the increasing shard iterator age. Increasing the shard count from 5 to 20 directly increases the number of Kinesis Data Streams shards, which in turn increases the number of concurrent Lambda invocations (one per shard) and the overall throughput of the stream. This addresses the bottleneck at the source without adding unnecessary complexity or cost, as KDS pricing is based on shard hours and Lambda concurrency is already set to 1000.

Answer

Increase the Lambda reserved concurrency to 5000 to allow more parallel executions.

Answer

Increase the batch window to 300 seconds to accumulate more records per invocation and reduce the number of calls.

Answer

Switch to using Amazon Kinesis Data Analytics with a larger instance type to process the stream.

Question 15

Drag and drop the steps to set up Amazon SageMaker Ground Truth for a labeling job in the correct order.

Accepted Answer

Prepare dataset, Create labeling job, Configure task settings, Provide labeling instructions, Execute labeling job. Ground Truth setup involves dataset preparation, job creation, task configuration, instructions, and execution.

Answer

Prepare dataset, Create labeling job, Provide labeling instructions, Configure task settings, Execute labeling job

Answer

Create labeling job, Prepare dataset, Configure task settings, Provide labeling instructions, Execute labeling job

Answer

Prepare dataset, Create labeling job, Configure task settings, Execute labeling job, Provide labeling instructions

Question 16

Match each AWS service to its primary purpose in a machine learning pipeline.

Accepted Answer

Amazon SageMaker: Fully managed service for building, training, and deploying machine learning models.. The correct matches associate each service with its primary ML pipeline role. SageMaker is for end-to-end ML, Lambda for serverless inference, S3 for data/model storage, Glue for data preparation. Distractors confuse these roles.

Answer

Amazon SageMaker: Serverless compute service for inference.

Answer

AWS Lambda: Scalable object storage for models.

Question 17

Match each data format to its typical use in AWS ML.

Accepted Answer

CSV: Tabular data for built-in algorithms. Common data formats in AWS ML: CSV for tabular data, JSON for semi-structured inference I/O, Parquet for columnar analytics, and TFRecord for TensorFlow training. Distractors confuse CSV with Parquet and JSON with binary formats.

Answer

CSV: Columnar storage for analytics

Answer

JSON: Binary format for efficient I/O

Question 18

A company uses AWS Glue ETL jobs to transform CSV data from an S3 bucket into Parquet. The jobs often fail with memory errors when processing large datasets. They want to minimize cost and improve reliability. What should they do?

Accepted Answer

Use G.1X or G.2X worker types and increase the number of DPUs per worker.. The G.1X and G.2X worker types provide more memory per worker (16 GB and 32 GB, respectively) compared to the standard G.0X worker (4 GB). By using these worker types and increasing the number of DPUs per worker, you allocate more memory to each task, reducing out-of-memory errors when processing large datasets. This approach also optimizes cost by using fewer, more powerful workers instead of many underpowered ones, improving reliability without unnecessary scaling.

Answer

Use Amazon Athena with CTAS queries to convert the data to Parquet.

Answer

Switch to S3 Batch Operations with AWS Lambda to process the files individually.

Answer

Increase the number of workers in the Glue job configuration.

Question 19

A machine learning team needs to create a training dataset by joining two large datasets (10 TB and 5 TB) stored in S3. The join key is 'user_id'. They want to minimize data movement and cost. Which approach should they use?

Accepted Answer

Use AWS Glue ETL to read both datasets, join them using Spark DataFrames, and write the result to S3.. AWS Glue ETL provides a serverless Spark environment that can read both datasets from S3, perform the join in a distributed manner, and write the result back to S3. This approach minimizes data movement because Glue reads directly from S3 and writes back to S3 without requiring intermediate storage or cluster management. It is cost-effective due to per-second billing and is optimized for large-scale ETL workloads like joining 10 TB and 5 TB datasets. Amazon Athena would scan 15 TB of data, incurring high costs and potential performance issues, while EMR and Redshift require provisioning and managing clusters, leading to higher operational overhead and costs.

Answer

Launch an Amazon EMR cluster with Spark, read data from S3, perform the join, and write results back to S3.

Answer

Use Amazon Athena to run a SQL query joining the two datasets directly on S3.

Answer

Load both datasets into Amazon Redshift using COPY commands, then perform the join in Redshift.

Question 20

A company uses an Amazon SageMaker notebook to train a model using data from an S3 bucket. The IAM role attached to the notebook has the following policy. What is the MOST specific change needed to allow the notebook to read from the bucket 'ml-data-123'?

Accepted Answer

Remove the Deny statement from the IAM policy.. The existing IAM policy includes an explicit Deny statement that blocks all s3:GetObject access to the bucket 'ml-data-123'. In IAM, an explicit Deny overrides any Allow, so even if other policies grant read access, the Deny prevents it. Removing the Deny statement is the most specific change because it eliminates the blocking condition without requiring additional permissions or resources.

Answer

Add an Allow statement for 's3:GetObject' on 'ml-data-123' to the IAM policy.

Answer

Create an S3 access point and update the IAM policy to use the access point ARN.

Answer

Add a bucket policy on 'ml-data-123' that grants access to the notebook's IAM role.