MLS-C01 Data Engineering — 50 Questions

Question 1

A data science team uses Amazon SageMaker to train models on a large dataset stored in S3. The dataset is 500 GB in CSV format and is updated daily. The team wants to optimize data loading for training jobs to reduce I/O wait time. Which data ingestion strategy is MOST effective?

Accepted Answer

Use SageMaker Pipe input mode to stream data directly from S3.. Option B is correct because SageMaker Pipe input mode streams data directly from S3 to the training algorithm without writing to the instance's EBS volume, eliminating disk I/O bottlenecks. This is especially effective for large datasets (500 GB) that are updated daily, as it reduces startup time and avoids the need to download the entire dataset before training begins.

Answer

Use SageMaker File input mode and increase the EBS volume size to 1 TB.

Answer

Convert the CSV files to Parquet format and use File input mode.

Answer

Load the data into an Amazon EFS file system and mount it to the training instance.

Question 2

A company uses Amazon Kinesis Data Streams to ingest real-time clickstream data from a website. The data is consumed by a Lambda function that writes records to an S3 bucket. Recently, the number of shards was increased from 2 to 4 to handle higher throughput. After the change, the Lambda function started processing records with increased latency and some records were being written out of order. What is the MOST likely cause?

Accepted Answer

The partition key used by the producer does not ensure that related records go to the same shard after resharding.. Option D is correct because after resharding from 2 to 4 shards, the mapping of partition keys to shards changes. If the producer does not use a partition key that ensures related records (e.g., same user session) are routed to the same shard, records that were previously ordered within a shard may now be split across multiple shards. Since the Lambda consumer processes shards independently, records from the same logical sequence can arrive out of order, and the increased shard count can also cause higher latency if the consumer is not properly parallelized.

Answer

The S3 bucket is not configured with versioning, causing overwrites.

Answer

The Lambda function is reading from the oldest sequence number, causing high IteratorAgeSeconds.

Answer

The Lambda function’s reserved concurrency is too low for the increased shard count.

Question 3

A data engineer needs to transform large CSV files stored in S3 into Parquet format and load them into a data warehouse for analysis. The transformation must be cost-effective and serverless. Which AWS service should be used?

Accepted Answer

AWS Glue. AWS Glue is the correct choice because it provides a fully managed, serverless ETL service that can automatically convert CSV files from S3 into Parquet format using its built-in Spark engine. It is cost-effective as you only pay for the resources consumed during the job execution, and it integrates directly with data warehouses like Amazon Redshift for loading transformed data.

Answer

Amazon Athena

Answer

Amazon EMR with Spark

Answer

AWS Data Pipeline

Question 4

A company uses Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data is JSON and must be partitioned by year, month, and day. The delivery stream is configured with a buffer interval of 60 seconds and buffer size of 5 MB. The data producer sends about 1 MB per second. The data is arriving in S3 but the partitions are not being created as expected. What is the MOST likely reason?

Accepted Answer

The delivery stream does not have dynamic partitioning enabled with the appropriate custom prefix.. Option B is correct because Kinesis Data Firehose requires dynamic partitioning to be explicitly enabled and configured with a custom prefix (e.g., 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/') to automatically partition data by year, month, and day. Without this setting, Firehose writes all data to a single S3 prefix, ignoring the desired partition structure.

Answer

The data is encrypted with AWS KMS and Firehose cannot write to encrypted buckets.

Answer

The buffer interval is too short for the data volume, causing incomplete records.

Answer

The S3 bucket has versioning enabled, which prevents partitioning.

Question 5

An ML team is building a recommendation system. The training data includes user-item interactions stored in Amazon DynamoDB. The team wants to export this data to S3 in Parquet format for use with Amazon SageMaker. The export should be incremental (only new or changed records) and run daily. Which approach meets these requirements with MINIMAL operational overhead?

Accepted Answer

Use DynamoDB Streams with AWS Lambda to write changes to S3 in Parquet format.. Option B is correct because DynamoDB Streams capture every change (insert, update, delete) in near real-time, and AWS Lambda can process these events to write only the changed records to S3 in Parquet format. This approach provides incremental, daily exports with minimal operational overhead, as it is fully serverless and requires no infrastructure management.

Answer

Use the DynamoDB Export to S3 feature and schedule it daily with AWS Glue.

Answer

Use a script that scans the DynamoDB table and filters by last updated timestamp.

Answer

Set up an Amazon EMR cluster running Spark jobs to read DynamoDB and write to S3.

Question 6

A data scientist uses Amazon SageMaker to train a model. The training dataset is 10 GB and stored in S3. The training job uses a ml.m5.large instance. The data must be available on the local file system during training. Which input mode should be used?

Accepted Answer

File input mode. File input mode is correct because it downloads the entire training dataset from S3 to the local file system of the ml.m5.large instance before training begins, ensuring the data is available locally as required. This mode is suitable for datasets up to 10 GB, as the instance's local storage (typically 8 GB for ml.m5.large) may be insufficient, but SageMaker uses the instance's Amazon EBS volume (up to 512 GB) for file input mode, making it viable.

Answer

Local input mode

Answer

Batch input mode

Answer

Pipe input mode

Question 7

A company uses AWS Glue ETL jobs to process data from multiple sources. The job fails with the error: 'An error occurred while calling o123.pyWriteDynamicFrame. Insufficient memory.' The job runs on a G.1X worker type with 10 workers. What should be changed to resolve this error?

Accepted Answer

Increase the number of workers to 20.. The error 'Insufficient memory' in AWS Glue ETL jobs typically indicates that the total memory across all executors is insufficient for the data being processed. Increasing the number of workers from 10 to 20 doubles the total memory and compute capacity available, allowing the job to handle larger datasets without running out of memory. This is the most direct and effective fix for a memory exhaustion error when using the G.1X worker type.

Answer

Enable the Spark UI to monitor the job.

Answer

Change the worker type to G.2X.

Answer

Reduce the number of partitions in the DynamicFrame.

Question 8

A company uses Amazon Redshift as a data warehouse. They need to load 50 TB of clickstream data from S3 into Redshift daily. The data arrives in 5-minute intervals as gzipped CSV files. The target table has a sort key and a distribution key. The load must complete within 2 hours. Which approach is MOST efficient?

Accepted Answer

Use the COPY command with a manifest file and gzip compression.. The COPY command is the most efficient way to load large volumes of data into Amazon Redshift because it uses the cluster's massively parallel processing (MPP) architecture to read data directly from S3 in parallel across all nodes. With a manifest file, you can specify multiple gzipped CSV files, and the gzip compression reduces network I/O and storage overhead. This approach can easily load 50 TB within 2 hours, especially when the target table has a sort key and distribution key, as COPY automatically leverages these for optimal data distribution and sorting during the load.

Answer

Use AWS Glue to transform the data and write to Redshift using JDBC.

Answer

Use a staging table and then merge using a stored procedure.

Answer

Use a series of INSERT statements from a Lambda function.

Question 9

A machine learning engineer needs to process a large dataset that does not fit on a single Amazon SageMaker notebook instance's EBS volume. The data is stored in S3. What is the MOST efficient way to access the data from the notebook?

Accepted Answer

Read the data directly from S3 using the boto3 library.. Option C is correct because reading data directly from S3 using the boto3 library is the most efficient approach for a dataset that exceeds the notebook instance's EBS volume capacity. Boto3 allows you to stream data in chunks or use S3 Select for server-side filtering, avoiding the need to download the entire dataset to local storage. This method leverages S3's high-throughput API and eliminates the bottleneck of writing to a local EBS volume, which is limited in size and I/O performance.

Answer

Increase the EBS volume size to 5 TB.

Answer

Mount the S3 bucket as a file system using s3fs.

Answer

Use SageMaker File input mode in the notebook.

Question 10

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to a sink. The application is failing with an 'OutOfMemoryError'. The application has parallelism set to 4 and uses 1 Kinesis Processing Unit (KPU). What is the MOST likely cause and solution?

Accepted Answer

The heap memory per operator is too low; increase parallelism to 8.. With parallelism set to 4 but only 1 KPU, each operator slot receives a fraction of the available heap memory, leading to an OutOfMemoryError. Increasing parallelism to 8 distributes the workload across more slots, but more importantly, it forces Kinesis Data Analytics to allocate additional KPUs (each KPU provides 4 GB of memory), thereby increasing the total heap memory available to the application.

Answer

The application is using too many operators; reduce parallelism to 2.

Answer

The checkpoint interval is too short; increase it to 5 minutes.

Answer

The buffer timeout is too high; reduce it to 50 ms.

Question 11

An organization stores sensitive customer data in S3. A data pipeline uses AWS Glue to transform the data and load it into Amazon Redshift. The security team requires that data be encrypted at rest in S3 and in transit between S3 and Glue, and between Glue and Redshift. Which configuration meets these requirements?

Accepted Answer

Enable S3 default encryption with SSE-KMS, use a VPC endpoint for S3, and configure Glue to use SSL for Redshift connection.. Option D is correct because it ensures encryption at rest in S3 via SSE-KMS, encrypts data in transit between S3 and Glue by using a VPC endpoint (which enforces HTTPS/TLS), and encrypts data in transit between Glue and Redshift by configuring SSL for the Redshift connection. SSE-KMS provides envelope encryption with a customer-managed key, while the VPC endpoint and SSL satisfy the in-transit encryption requirements.

Answer

Use S3 client-side encryption, and use VPC Peering between Glue and Redshift.

Answer

Use S3 default encryption with SSE-KMS, and use Network Load Balancer for Redshift.

Answer

Enable S3 server-side encryption with SSE-S3, and use SSL for both Glue connections.

Question 12

A data science team is building a real-time fraud detection system. Transactions are streamed via Amazon Kinesis Data Streams, and a Lambda function performs feature engineering and invokes an Amazon SageMaker endpoint for predictions. The team notices that the Lambda function is timing out and causing data loss. Which solution should the team implement to process the stream reliably and at low latency?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to consume the stream, perform feature engineering, and invoke the SageMaker endpoint with exactly-once processing.. Option A is correct because Amazon Kinesis Data Analytics for Apache Flink provides a stateful, low-latency stream processing engine that can consume from Kinesis Data Streams, perform feature engineering in real-time, and invoke SageMaker endpoints with exactly-once processing semantics. This eliminates Lambda timeouts and data loss by using a long-running, scalable application instead of a short-lived function.

Answer

Use the Kinesis Client Library (KCL) to process the stream in an Amazon EC2 instance, and store the predictions in Amazon DynamoDB.

Answer

Increase the Lambda function timeout to 15 minutes and allocate more memory to reduce processing time.

Answer

Configure Amazon Kinesis Firehose to deliver the stream to an Amazon S3 bucket, then trigger a Lambda function to process the data in batches.

Question 13

A company uses Amazon SageMaker to train and deploy machine learning models. The training data is stored in Amazon S3 (Parquet format, 10 TB). The data scientists have been running training jobs using the File mode input, but the jobs are taking too long due to data download time. They want to reduce the training start-up time and overall training time. Which solution is MOST cost-effective and efficient?

Accepted Answer

Configure the SageMaker training job to use Pipe mode, which streams data directly from S3 without downloading to the instance's local storage.. Pipe mode in SageMaker streams training data directly from Amazon S3 to the training algorithm without first downloading it to the instance's local storage. This eliminates the data download step, significantly reducing startup time and overall training time for large datasets like 10 TB. It is the most cost-effective because it avoids the need for larger instances or additional data transfer acceleration services.

Answer

Use S3 Transfer Acceleration to speed up the data transfer from S3 to the training instance.

Answer

Use larger EC2 instances with more vCPUs and memory to speed up the training process.

Answer

Enable Elastic Fabric Adapter (EFA) on the training instances to improve network throughput.

Question 14

A data engineer is building a data pipeline to process user clickstream data. The data arrives as JSON files in an S3 bucket. The pipeline must transform the JSON into Parquet format and partition by date and event type, then make the data available for Amazon Athena queries. The engineer needs a fully managed, serverless solution with minimal operational overhead. Which combination of AWS services should the engineer use?

Accepted Answer

Use S3 Event Notifications to trigger an AWS Lambda function that converts the JSON to Parquet and writes to a partitioned S3 location, then query with Athena.. Option C is correct because AWS Lambda triggered by S3 Event Notifications provides a fully serverless, event-driven architecture with minimal operational overhead for converting JSON to Parquet and partitioning by date and event type. Lambda can process each new JSON file as it arrives, perform the transformation in memory (using libraries like PyArrow or Pandas), and write the Parquet output to a partitioned S3 path, which Athena can then query directly. This approach avoids managing any clusters or job scheduling, aligning with the requirement for a fully managed, serverless solution.

Answer

Use Amazon EMR with Spark to read JSON, convert to Parquet, and partition, then query with Athena.

Answer

Use AWS Glue ETL jobs to read JSON from S3, transform to Parquet, and write to a partitioned S3 location, then use Athena.

Answer

Use Amazon Kinesis Firehose to ingest data and convert to Parquet, then write to S3, and query with Athena.

Question 15

A team is using Amazon SageMaker to train a model on a dataset that is 500 GB in size, stored as CSV files in S3. The training job takes 2 hours using a single ml.p3.2xlarge instance. The team wants to reduce training time to under 30 minutes. The model architecture supports distributed training. Which solution will achieve this goal with the LEAST amount of code changes?

Accepted Answer

Use multiple ml.p3.2xlarge instances with SageMaker's distributed data parallelism library, enabling automatic sharding of the training data.. Option C is correct because SageMaker's distributed data parallelism library automatically shards the training data across multiple ml.p3.2xlarge instances, enabling parallel gradient computation and reducing wall-clock training time from 2 hours to under 30 minutes without requiring manual code changes to the training script. The model architecture already supports distributed training, so the library handles the communication and synchronization (e.g., AllReduce) transparently.

Answer

Use managed spot training to reduce cost and then use cost savings to train with a larger instance.

Answer

Use a single ml.p3.16xlarge instance with more GPUs and memory.

Answer

Change the input mode to Pipe mode to stream data from S3 directly, reducing I/O wait time.

Question 16

A company processes large streams of IoT sensor data using Amazon Kinesis Data Streams with 100 shards. Each sensor reading is about 1 KB. The data is consumed by an Amazon EMR cluster running Spark Streaming jobs. The team notices that the Spark Streaming job's processing time is gradually increasing, and the stream is falling behind. They suspect the issue is due to skewed data distribution across shards. Which approach should the team take to diagnose and resolve the issue?

Accepted Answer

Modify the producer to add a random prefix to the partition key, ensuring even distribution across all shards, and monitor the stream using CloudWatch.. Option B is correct because adding a random prefix to the partition key ensures that sensor data is evenly distributed across all 100 shards, eliminating hot shards that cause processing delays. This directly addresses the skewed data distribution issue without requiring infrastructure changes, and the team can monitor the improvement using CloudWatch metrics like IncomingBytes and ReadProvisionedThroughputExceeded.

Answer

Increase the number of shards to 200 to provide more parallelism.

Answer

Check Amazon CloudWatch metrics for Kinesis to identify hot shards, then manually redistribute the data by repartitioning in Spark.

Answer

Use the Kinesis Client Library (KCL) with a custom worker to rebalance the load across shards.

Question 17

A data engineering team is designing a data lake on AWS for machine learning workloads. The data includes structured, semi-structured, and unstructured data. The team needs to ensure that the data is cataloged, easily discoverable, and can be queried by Amazon Athena and Amazon EMR. The team also wants to enforce fine-grained access control at the column and row level for sensitive data. Which combination of AWS services should the team use? (Select TWO.)

Accepted Answer

AWS Lake Formation. AWS Lake Formation is correct because it provides a centralized service to build, secure, and manage data lakes on AWS. It enables fine-grained access control at the column and row level for sensitive data, which directly meets the requirement for enforcing such controls. Additionally, Lake Formation integrates with Amazon Athena and Amazon EMR for querying and processing the cataloged data.

Answer

AWS Identity and Access Management (IAM)

Answer

Amazon RDS for PostgreSQL

Answer

Amazon DynamoDB

Question 18

A company is building a real-time anomaly detection system for network traffic logs. The logs are ingested via Amazon Kinesis Data Streams and processed with an Amazon SageMaker endpoint for inference. The team needs to ensure that the inference results are stored durably and can be replayed for model retraining. The system must handle at least 10,000 records per second with low latency. Which three AWS services should the team use to build this architecture? (Select THREE.)

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is the correct ingestion layer because it provides durable, real-time data streaming with the ability to handle over 10,000 records per second. It acts as the source of truth for network traffic logs, enabling low-latency processing and replay for model retraining.

Answer

AWS Glue ETL

Answer

Amazon Kinesis Data Firehose

Question 19

A data scientist needs to transform raw JSON data from an S3 bucket into Parquet format using AWS Glue. The job must be cost-effective and run only when new data arrives. Which solution should be used?

Accepted Answer

Create a Glue ETL job triggered by an S3 event notification via Lambda.. Option D is correct because it uses an S3 event notification to invoke a Lambda function, which then triggers an AWS Glue ETL job only when new data arrives. This event-driven architecture ensures cost-effectiveness by avoiding continuous or scheduled runs, and it directly transforms raw JSON into Parquet format as required.

Answer

Create a Glue crawler that runs continuously.

Answer

Schedule a Glue ETL job to run every hour.

Answer

Use Glue DataBrew to transform data and schedule it daily.

Question 20

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Lambda function that writes to an S3 bucket. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors. What is the MOST likely cause?

Accepted Answer

The Kinesis stream has too few shards for the data volume.. The 'ProvisionedThroughputExceededException' error in Amazon Kinesis Data Streams indicates that the data ingestion rate exceeds the write capacity of the stream's shards. Each shard supports up to 1 MB/s or 1,000 records/s for writes. If the clickstream data volume surpasses this limit, the Lambda function, which reads from the stream, will encounter this exception. Increasing the number of shards scales the write capacity to match the data volume.

Answer

The data retention period of the stream is too short.

Answer

The S3 bucket has insufficient write capacity.

Answer

The Lambda function's reserved concurrency is set too high.