MLS-C01 Data Engineering — 10 Questions

Question 1

A data science team uses Amazon SageMaker to train models on a large dataset stored in S3. The dataset is 500 GB in CSV format and is updated daily. The team wants to optimize data loading for training jobs to reduce I/O wait time. Which data ingestion strategy is MOST effective?

Accepted Answer

Use SageMaker Pipe input mode to stream data directly from S3.. Option B is correct because SageMaker Pipe input mode streams data directly from S3 to the training algorithm without writing to the instance's EBS volume, eliminating disk I/O bottlenecks. This is especially effective for large datasets (500 GB) that are updated daily, as it reduces startup time and avoids the need to download the entire dataset before training begins.

Answer

Use SageMaker File input mode and increase the EBS volume size to 1 TB.

Answer

Convert the CSV files to Parquet format and use File input mode.

Answer

Load the data into an Amazon EFS file system and mount it to the training instance.

Question 2

A company uses Amazon Kinesis Data Streams to ingest real-time clickstream data from a website. The data is consumed by a Lambda function that writes records to an S3 bucket. Recently, the number of shards was increased from 2 to 4 to handle higher throughput. After the change, the Lambda function started processing records with increased latency and some records were being written out of order. What is the MOST likely cause?

Accepted Answer

The partition key used by the producer does not ensure that related records go to the same shard after resharding.. Option D is correct because after resharding from 2 to 4 shards, the mapping of partition keys to shards changes. If the producer does not use a partition key that ensures related records (e.g., same user session) are routed to the same shard, records that were previously ordered within a shard may now be split across multiple shards. Since the Lambda consumer processes shards independently, records from the same logical sequence can arrive out of order, and the increased shard count can also cause higher latency if the consumer is not properly parallelized.

Answer

The S3 bucket is not configured with versioning, causing overwrites.

Answer

The Lambda function is reading from the oldest sequence number, causing high IteratorAgeSeconds.

Answer

The Lambda function’s reserved concurrency is too low for the increased shard count.

Question 3

A data engineer needs to transform large CSV files stored in S3 into Parquet format and load them into a data warehouse for analysis. The transformation must be cost-effective and serverless. Which AWS service should be used?

Accepted Answer

AWS Glue. AWS Glue is the correct choice because it provides a fully managed, serverless ETL service that can automatically convert CSV files from S3 into Parquet format using its built-in Spark engine. It is cost-effective as you only pay for the resources consumed during the job execution, and it integrates directly with data warehouses like Amazon Redshift for loading transformed data.

Answer

Amazon Athena

Answer

Amazon EMR with Spark

Answer

AWS Data Pipeline

Question 4

A company uses Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data is JSON and must be partitioned by year, month, and day. The delivery stream is configured with a buffer interval of 60 seconds and buffer size of 5 MB. The data producer sends about 1 MB per second. The data is arriving in S3 but the partitions are not being created as expected. What is the MOST likely reason?

Accepted Answer

The delivery stream does not have dynamic partitioning enabled with the appropriate custom prefix.. Option B is correct because Kinesis Data Firehose requires dynamic partitioning to be explicitly enabled and configured with a custom prefix (e.g., 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/') to automatically partition data by year, month, and day. Without this setting, Firehose writes all data to a single S3 prefix, ignoring the desired partition structure.

Answer

The data is encrypted with AWS KMS and Firehose cannot write to encrypted buckets.

Answer

The buffer interval is too short for the data volume, causing incomplete records.

Answer

The S3 bucket has versioning enabled, which prevents partitioning.

Question 5

An ML team is building a recommendation system. The training data includes user-item interactions stored in Amazon DynamoDB. The team wants to export this data to S3 in Parquet format for use with Amazon SageMaker. The export should be incremental (only new or changed records) and run daily. Which approach meets these requirements with MINIMAL operational overhead?

Accepted Answer

Use DynamoDB Streams with AWS Lambda to write changes to S3 in Parquet format.. Option B is correct because DynamoDB Streams capture every change (insert, update, delete) in near real-time, and AWS Lambda can process these events to write only the changed records to S3 in Parquet format. This approach provides incremental, daily exports with minimal operational overhead, as it is fully serverless and requires no infrastructure management.

Answer

Use the DynamoDB Export to S3 feature and schedule it daily with AWS Glue.

Answer

Use a script that scans the DynamoDB table and filters by last updated timestamp.

Answer

Set up an Amazon EMR cluster running Spark jobs to read DynamoDB and write to S3.

Question 6

A data scientist uses Amazon SageMaker to train a model. The training dataset is 10 GB and stored in S3. The training job uses a ml.m5.large instance. The data must be available on the local file system during training. Which input mode should be used?

Accepted Answer

File input mode. File input mode is correct because it downloads the entire training dataset from S3 to the local file system of the ml.m5.large instance before training begins, ensuring the data is available locally as required. This mode is suitable for datasets up to 10 GB, as the instance's local storage (typically 8 GB for ml.m5.large) may be insufficient, but SageMaker uses the instance's Amazon EBS volume (up to 512 GB) for file input mode, making it viable.

Answer

Local input mode

Answer

Batch input mode

Answer

Pipe input mode

Question 7

A company uses AWS Glue ETL jobs to process data from multiple sources. The job fails with the error: 'An error occurred while calling o123.pyWriteDynamicFrame. Insufficient memory.' The job runs on a G.1X worker type with 10 workers. What should be changed to resolve this error?

Accepted Answer

Increase the number of workers to 20.. The error 'Insufficient memory' in AWS Glue ETL jobs typically indicates that the total memory across all executors is insufficient for the data being processed. Increasing the number of workers from 10 to 20 doubles the total memory and compute capacity available, allowing the job to handle larger datasets without running out of memory. This is the most direct and effective fix for a memory exhaustion error when using the G.1X worker type.

Answer

Enable the Spark UI to monitor the job.

Answer

Change the worker type to G.2X.

Answer

Reduce the number of partitions in the DynamicFrame.

Question 8

A company uses Amazon Redshift as a data warehouse. They need to load 50 TB of clickstream data from S3 into Redshift daily. The data arrives in 5-minute intervals as gzipped CSV files. The target table has a sort key and a distribution key. The load must complete within 2 hours. Which approach is MOST efficient?

Accepted Answer

Use the COPY command with a manifest file and gzip compression.. The COPY command is the most efficient way to load large volumes of data into Amazon Redshift because it uses the cluster's massively parallel processing (MPP) architecture to read data directly from S3 in parallel across all nodes. With a manifest file, you can specify multiple gzipped CSV files, and the gzip compression reduces network I/O and storage overhead. This approach can easily load 50 TB within 2 hours, especially when the target table has a sort key and distribution key, as COPY automatically leverages these for optimal data distribution and sorting during the load.

Answer

Use AWS Glue to transform the data and write to Redshift using JDBC.

Answer

Use a staging table and then merge using a stored procedure.

Answer

Use a series of INSERT statements from a Lambda function.

Question 9

A machine learning engineer needs to process a large dataset that does not fit on a single Amazon SageMaker notebook instance's EBS volume. The data is stored in S3. What is the MOST efficient way to access the data from the notebook?

Accepted Answer

Read the data directly from S3 using the boto3 library.. Option C is correct because reading data directly from S3 using the boto3 library is the most efficient approach for a dataset that exceeds the notebook instance's EBS volume capacity. Boto3 allows you to stream data in chunks or use S3 Select for server-side filtering, avoiding the need to download the entire dataset to local storage. This method leverages S3's high-throughput API and eliminates the bottleneck of writing to a local EBS volume, which is limited in size and I/O performance.

Answer

Increase the EBS volume size to 5 TB.

Answer

Mount the S3 bucket as a file system using s3fs.

Answer

Use SageMaker File input mode in the notebook.

Question 10

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to a sink. The application is failing with an 'OutOfMemoryError'. The application has parallelism set to 4 and uses 1 Kinesis Processing Unit (KPU). What is the MOST likely cause and solution?

Accepted Answer

The heap memory per operator is too low; increase parallelism to 8.. With parallelism set to 4 but only 1 KPU, each operator slot receives a fraction of the available heap memory, leading to an OutOfMemoryError. Increasing parallelism to 8 distributes the workload across more slots, but more importantly, it forces Kinesis Data Analytics to allocate additional KPUs (each KPU provides 4 GB of memory), thereby increasing the total heap memory available to the application.

Answer

The application is using too many operators; reduce parallelism to 2.

Answer

The checkpoint interval is too short; increase it to 5 minutes.

Answer

The buffer timeout is too high; reduce it to 50 ms.