Courseiva

CCNA Ml Data Engineering Questions

50 of 350 questions · Page 5/5 · Ml Data Engineering topic · Answers revealed

Practice these questionsExam hubAll questions
301
MCQhard

A retail company runs an e-commerce platform on AWS. They have a Data Engineering team that processes clickstream data using Amazon Kinesis Data Streams (KDS) with a shard count of 5. The data is consumed by an AWS Lambda function that transforms and loads the data into an Amazon S3 bucket partitioned by year/month/day/hour. Recently, the team has noticed that the Lambda function is experiencing throttling errors, and the KDS shard iterator age is increasing, indicating that the consumer cannot keep up with the incoming data rate. The team has already increased the Lambda reserved concurrency to 1000 and enabled batch window of 60 seconds. The metrics show that the Lambda function duration is well under the 5-minute timeout, and there are no errors in the transformation logic. The S3 write operations are not failing. Which course of action would MOST effectively resolve the issue without unnecessary cost or complexity?

A.Increase the number of shards in the Kinesis Data Stream to 20 to increase the parallelism of Lambda consumers.
B.Increase the Lambda reserved concurrency to 5000 to allow more parallel executions.
C.Increase the batch window to 300 seconds to accumulate more records per invocation and reduce the number of calls.
D.Switch to using Amazon Kinesis Data Analytics with a larger instance type to process the stream.
AnswerA

More shards allow more concurrent Lambda invocations, improving throughput and reducing iterator age.

Why this answer

The core issue is that the Lambda consumer cannot keep up with the incoming data rate, as evidenced by the increasing shard iterator age. Increasing the shard count from 5 to 20 directly increases the number of Kinesis Data Streams shards, which in turn increases the number of concurrent Lambda invocations (one per shard) and the overall throughput of the stream. This addresses the bottleneck at the source without adding unnecessary complexity or cost, as KDS pricing is based on shard hours and Lambda concurrency is already set to 1000.

Exam trap

The trap here is that candidates often assume increasing Lambda concurrency or batch window will solve throughput issues, but they fail to recognize that Kinesis shard count is the fundamental limiter of parallelism in the Lambda-Kinesis integration.

How to eliminate wrong answers

Option B is wrong because increasing Lambda reserved concurrency to 5000 does not help when the bottleneck is the number of Kinesis shards; Lambda can only process one shard per concurrent invocation, and with only 5 shards, the maximum parallelism is 5, so additional concurrency is unused. Option C is wrong because increasing the batch window to 300 seconds would increase latency and could cause the shard iterator age to grow further, as records would accumulate longer before being processed, worsening the backlog. Option D is wrong because switching to Kinesis Data Analytics introduces a different service (meant for real-time analytics with SQL or Flink) that adds complexity and cost, and does not directly address the consumer throughput limitation caused by insufficient shard parallelism.

Practice this question →
302
MCQhard

A company uses AWS Lake Formation to manage permissions on a data lake stored in Amazon S3. A data analyst tries to query a table using Amazon Athena but receives an 'Access Denied' error. The analyst has SELECT permission on the table in Lake Formation. What is the most likely cause?

A.The S3 bucket is not registered with Lake Formation
B.The S3 bucket is encrypted with a KMS key that the analyst does not have access to
C.The table does not have any partitions defined
D.The IAM role used by Athena does not have lakeformation:GetDataAccess permission
AnswerA

If the bucket is not registered, Lake Formation cannot control access, and the default S3 permissions apply, which may deny access.

Why this answer

When Lake Formation manages permissions on a data lake, it requires that the underlying S3 bucket be registered with Lake Formation. If the bucket is not registered, Lake Formation cannot enforce its fine-grained access controls, and Athena will fail with an 'Access Denied' error even if the analyst has SELECT permission on the table in Lake Formation. Registering the bucket allows Lake Formation to integrate with S3 and apply its permission model.

Exam trap

The trap here is that candidates often assume 'Access Denied' errors are always due to missing IAM permissions or encryption issues, but in Lake Formation, the most common root cause is the S3 bucket not being registered, which prevents Lake Formation from enforcing its permissions.

How to eliminate wrong answers

Option B is wrong because while KMS key access issues can cause 'Access Denied' errors, the question states the analyst has SELECT permission on the table in Lake Formation, and the most likely cause given the scenario is the missing bucket registration, not encryption. Option C is wrong because a table without partitions can still be queried (though it may be inefficient); missing partitions do not cause 'Access Denied' errors. Option D is wrong because the IAM role used by Athena does not need lakeformation:GetDataAccess permission; instead, Athena assumes a role that must have permissions to call Lake Formation APIs, and the error is more commonly due to the bucket not being registered with Lake Formation.

Practice this question →
303
MCQeasy

A data engineering team needs to set up a data pipeline that ingests streaming data from an Apache Kafka cluster running on Amazon EKS into an S3 data lake. The data must be stored in Parquet format, partitioned by date and event type. The team wants a fully managed solution with minimal operational overhead. Which solution should they choose?

A.Use Amazon MSK (Managed Streaming for Apache Kafka) and configure an MSK Connect S3 sink connector.
B.Set up a Kinesis Data Firehose delivery stream that reads from Kafka and writes to S3.
C.Use AWS Glue ETL jobs to pull data from Kafka cluster periodically.
D.Create a Kinesis Data Analytics application to read from Kafka and write to S3.
AnswerA

MSK is fully managed Kafka, and MSK Connect can stream data to S3 in Parquet format.

Why this answer

Amazon MSK is a fully managed Apache Kafka service that integrates with MSK Connect, which provides a pre-built S3 sink connector. This connector can directly stream data from Kafka topics to S3 in Parquet format with partitioning by date and event type, requiring no custom code or infrastructure management. This minimizes operational overhead while meeting all requirements for a fully managed solution.

Exam trap

The trap here is that candidates often confuse Kinesis Data Firehose's ability to accept data from various sources with direct Kafka integration, but Firehose does not natively support Kafka as a source without additional services like Kinesis Data Streams or a custom producer.

How to eliminate wrong answers

Option B is wrong because Kinesis Data Firehose cannot directly read from a Kafka cluster; it requires a Kinesis Data Streams or other sources, not Kafka. Option C is wrong because AWS Glue ETL jobs are batch-oriented and not designed for real-time streaming ingestion with minimal overhead; they require periodic polling and manual orchestration. Option D is wrong because Kinesis Data Analytics is intended for real-time analytics using SQL or Flink, not for direct data ingestion to S3; it would require additional components to write to S3, increasing complexity.

Practice this question →
304
Multi-Selectmedium

A data engineering team is designing a data pipeline to process streaming data from social media feeds. The data must be deduplicated, enriched with customer information from a relational database, and stored in Amazon S3 in Parquet format. Which AWS services should the team use to build this pipeline? (Select TWO.)

Select 2 answers
A.AWS Glue
B.Amazon Kinesis Data Firehose
C.Amazon Athena
D.Amazon SageMaker
E.Amazon Kinesis Data Streams
AnswersA, E

Glue ETL can transform and enrich data from streams and databases.

Why this answer

AWS Glue is correct because it provides a serverless ETL service that can transform streaming data stored in Amazon S3 into Parquet format. It can also connect to a relational database via JDBC to enrich the data with customer information, and its built-in deduplication capabilities (e.g., using DropDuplicates in PySpark) handle the deduplication requirement.

Exam trap

AWS often tests the distinction between data ingestion services (Kinesis Data Firehose) and data processing/ETL services (AWS Glue), leading candidates to mistakenly select Firehose for deduplication and enrichment tasks that it cannot natively perform.

Practice this question →
305
MCQeasy

An ML engineer is using Amazon SageMaker to train a model on a dataset that contains personal identifiable information (PII). The data must be encrypted at rest and in transit. The company uses AWS KMS for key management. How should the engineer configure the SageMaker training job to meet these encryption requirements?

A.Enable S3 Server-Side Encryption (SSE-S3) on the input data bucket
B.Use a custom Docker image with built-in encryption and disable inter-container traffic encryption for performance
C.Use a VPC with an S3 VPC Endpoint and enable SSL for the endpoint
D.Specify a KMS key for the training job's VolumeKmsKeyId and enable inter-container traffic encryption
AnswerD

This encrypts the ML storage volume and inter-container traffic.

Why this answer

It addresses both encryption at rest and in transit for the SageMaker training job. Specifying a KMS key via VolumeKmsKeyId encrypts the ML storage volume (EBS) used by the training instances at rest, while enabling inter-container traffic encryption ensures data exchanged between distributed training containers is encrypted in transit using TLS. This combination meets the PII encryption requirements using AWS KMS.

Exam trap

The trap here is that candidates often focus only on S3 encryption or VPC endpoints, overlooking that SageMaker training jobs have separate encryption requirements for local storage and inter-container communication, which are explicitly controlled by VolumeKmsKeyId and inter-container traffic encryption settings.

How to eliminate wrong answers

Option A is wrong because S3 Server-Side Encryption (SSE-S3) encrypts data at rest in S3 but does not encrypt the SageMaker training job's local storage volumes or inter-container traffic; it also does not use AWS KMS as required. Option B is wrong because using a custom Docker image with built-in encryption is unnecessary and does not leverage AWS KMS; disabling inter-container traffic encryption violates the encryption-in-transit requirement. Option C is wrong because a VPC with an S3 VPC Endpoint and SSL only secures the data transfer between SageMaker and S3, but does not encrypt the training job's EBS volumes at rest or inter-container traffic within the job.

Practice this question →
306
MCQmedium

A data scientist is training a deep learning model using a large dataset stored in S3. The training job runs on a SageMaker training instance with a GPU. The data engineer notices that the GPU utilization is low, and the training is I/O bound. The data is read directly from S3 using the SageMaker SDK. Which change should the data engineer recommend to improve GPU utilization?

A.Increase the batch size in the training script to process more data per step.
B.Mount the S3 bucket to the training instance using Amazon Elastic File System (EFS).
C.Use SageMaker Pipe mode to stream data directly from S3 to the training container.
D.Copy the entire dataset to an Amazon EBS volume attached to the training instance.
AnswerC

Pipe mode eliminates disk I/O, allowing data to be streamed directly to the GPU.

Why this answer

SageMaker Pipe mode streams data directly from S3 to the training container, eliminating the need to download the entire dataset to disk. This reduces I/O latency and keeps the GPU fed with data, improving utilization. The current I/O bottleneck occurs because the SDK reads data from S3 as files, causing the GPU to wait for data.

Exam trap

The trap here is that candidates often confuse 'mounting S3' (which is not natively supported without third-party tools like s3fs-fuse) with SageMaker's built-in Pipe mode, or they incorrectly assume that increasing batch size will compensate for slow data loading.

How to eliminate wrong answers

Option A is wrong because increasing batch size does not address the I/O bottleneck; it may even worsen memory pressure and does not speed up data ingestion from S3. Option B is wrong because mounting an S3 bucket via EFS is not a supported or efficient approach; EFS is a separate NFS-based file system, not a direct S3 mount, and would introduce additional latency. Option D is wrong because copying the entire dataset to an EBS volume adds significant startup time and storage cost, and does not solve the streaming data issue; it still requires a full download before training begins.

Practice this question →
307
MCQhard

A company has an AWS Glue ETL job that reads data from an Amazon RDS for MySQL table and writes to Amazon S3 in Parquet format. The job runs daily and processes 500 GB of data. Recently, the job has been failing with memory errors during the write phase. The data schema is wide (200 columns). Which change should a data engineer make to the Glue job to resolve the memory issue?

A.Increase the number of DPUs for the Glue job.
B.Change the output format from Parquet to CSV.
C.Use the JDBC connection with fetchSize parameter.
D.Configure the write operation with 'groupSize' to limit records per file.
AnswerD

Limiting records per file reduces the memory needed for buffering during writes.

Why this answer

The memory error occurs because the wide schema (200 columns) and large data volume (500 GB) cause the Spark executors to run out of memory when writing Parquet files, as each executor attempts to buffer entire partitions. Configuring 'groupSize' limits the number of records written per file, reducing the per-executor memory footprint and preventing out-of-memory errors during the write phase.

Exam trap

The trap here is that candidates often assume memory errors are solved by adding more resources (DPUs) or by changing the output format, when the actual fix is a write-tuning parameter that controls per-file record limits.

How to eliminate wrong answers

Option A is wrong because increasing DPUs adds more parallelism but does not reduce the per-executor memory pressure caused by wide rows and large partitions; it may even exacerbate memory issues by increasing shuffle overhead. Option B is wrong because changing to CSV would increase file size and I/O, and does not address the root cause of memory exhaustion during write buffering. Option C is wrong because the fetchSize parameter controls how many rows are fetched per JDBC round trip from MySQL, which affects read performance, not memory usage during the write phase to S3.

Practice this question →
308
MCQeasy

A data engineer needs to schedule an AWS Glue ETL job to run every hour. Which service should be used to trigger the job?

A.AWS Lambda
B.AWS Step Functions
C.Amazon CloudWatch Events (EventBridge)
D.AWS Data Pipeline
AnswerC

EventBridge can schedule cron jobs to trigger Glue.

Why this answer

Amazon CloudWatch Events (now part of Amazon EventBridge) is the correct service for scheduling AWS Glue ETL jobs on a recurring basis, such as every hour. It allows you to create a time-based rule using a cron or rate expression that triggers an AWS Glue job directly as a target, without needing additional compute or orchestration logic.

Exam trap

The trap here is that candidates often confuse AWS Lambda as a scheduler because it can be triggered by CloudWatch Events, but the question asks for the service that triggers the job, not the service that runs the trigger logic; EventBridge is the native scheduling service, while Lambda is a compute target.

How to eliminate wrong answers

Option A is wrong because AWS Lambda is a serverless compute service for running code in response to events, not a scheduling service; while you could use Lambda to invoke Glue, it adds unnecessary complexity and cost compared to a native scheduled trigger. Option B is wrong because AWS Step Functions is a workflow orchestration service designed for coordinating multiple AWS services and handling state, not for simple time-based scheduling; using it for a single hourly trigger would be over-engineering. Option D is wrong because AWS Data Pipeline is a batch data processing and orchestration service that is more heavyweight and designed for complex data workflows with dependencies, not for straightforward hourly job scheduling; it also requires managing pipeline definitions and resources that are unnecessary for this use case.

Practice this question →
309
MCQhard

You are a data engineer at a fintech company. The company processes real-time stock market data from multiple exchanges. The data is ingested via Amazon Kinesis Data Streams with 50 shards. Each record is about 1 KB, and the ingestion rate is 5,000 records per second. The data is consumed by a Java application running on Amazon ECS that performs real-time analytics and stores results in Amazon DynamoDB. Recently, the application has been experiencing high latency, and some records are stuck in the shards for minutes before being consumed. The CloudWatch metrics show that the application's CPU utilization is low, but the iterator age is increasing. The application uses the Kinesis Client Library (KCL) with a single worker. What is the most likely cause and how should it be fixed?

A.Increase the number of shards to 200 to provide more throughput.
B.Increase the CPU capacity of the ECS task by moving to a larger instance type.
C.Move the destination from DynamoDB to Amazon RDS to reduce write latency.
D.Scale the number of KCL workers to match the number of shards (e.g., 50 workers) to process shards in parallel.
AnswerD

A single worker can only process one shard at a time; with 50 shards, records in other shards wait. Multiple workers can process shards concurrently, reducing latency.

Why this answer

A single KCL worker processes all shards sequentially, causing high iterator age with 50 shards. Scaling to 50 workers (one per shard) enables parallel processing, reducing latency. Option A is incorrect because 50 shards provide up to 50 MB/s write capacity, far exceeding the actual ~5 MB/s (5000 records/sec * 1 KB).

Option B is incorrect because CPU utilization is low, indicating the bottleneck is not compute but parallelization. Option C is incorrect because DynamoDB write latency is not the issue; the problem is ingestion-side processing delay.

Practice this question →
310
MCQmedium

A data science team needs to process streaming data from thousands of IoT devices and perform real-time anomaly detection. The data must be persisted in Amazon S3 for batch processing later. Which combination of AWS services should be used to meet these requirements?

A.Amazon Kinesis Data Streams for ingestion, Amazon Kinesis Data Analytics for anomaly detection, and Amazon Kinesis Data Firehose to deliver data to Amazon S3.
B.Amazon Kinesis Data Streams for ingestion, AWS Glue for anomaly detection, and Amazon S3 for storage.
C.AWS Lambda for both ingestion and anomaly detection, and Amazon S3 for storage.
D.Amazon Simple Queue Service (SQS) for ingestion, AWS Lambda for anomaly detection, and Amazon S3 for storage.
AnswerA

This combination provides real-time ingestion, analytics, and durable storage.

Why this answer

Amazon Kinesis Data Streams provides durable, real-time ingestion for high-throughput IoT data. Kinesis Data Analytics can perform SQL-based anomaly detection on the stream, and Kinesis Data Firehose reliably delivers the processed or raw data to Amazon S3 for batch processing. This combination meets all requirements for streaming ingestion, real-time analytics, and persistent storage.

Exam trap

The trap here is that candidates may confuse AWS Glue's batch processing capabilities with real-time streaming analytics, or assume Lambda can handle continuous high-throughput ingestion without considering its timeout and scaling limitations.

How to eliminate wrong answers

Option B is wrong because AWS Glue is a batch ETL service, not designed for real-time anomaly detection on streaming data. Option C is wrong because AWS Lambda has a maximum execution timeout of 15 minutes and is not suitable for continuous high-throughput ingestion from thousands of devices, nor does it natively support persistent streaming state for anomaly detection. Option D is wrong because Amazon SQS is a message queue for decoupled communication, not a streaming ingestion service, and it lacks the ordering and replay capabilities needed for real-time anomaly detection on IoT data streams.

Practice this question →
311
MCQmedium

A machine learning team needs to process a large dataset stored in Amazon S3 using Apache Spark. They want to minimize cost and avoid managing infrastructure. Which AWS service should they use?

A.AWS Glue
B.Amazon Athena
C.Amazon EMR
D.Amazon SageMaker
AnswerA

Glue provides serverless Spark for ETL on S3 data.

Why this answer

AWS Glue is a fully managed, serverless Spark environment that eliminates infrastructure management. It directly meets the requirement to process large datasets in S3 with Apache Spark while minimizing cost, as you only pay for resources consumed during job execution. Glue also integrates natively with S3 and the broader AWS ecosystem, making it the optimal choice for this use case.

Exam trap

The trap here is that candidates often confuse Amazon EMR's managed cluster feature with 'serverless,' but EMR still requires provisioning and managing EC2 instances, whereas AWS Glue is truly serverless and infrastructure-free.

How to eliminate wrong answers

Option B (Amazon Athena) is wrong because it is a serverless query service for SQL-based analysis, not a platform for running Apache Spark jobs. Option C (Amazon EMR) is wrong because, while it supports Spark, it requires managing EC2 clusters and infrastructure, contradicting the 'avoid managing infrastructure' requirement. Option D (Amazon SageMaker) is wrong because it is a machine learning platform for building, training, and deploying models, not a general-purpose Spark processing service.

Practice this question →
312
Matchingmedium

Match each AWS service to its primary purpose in a machine learning pipeline.

Drag a concept onto its matching description — or click a concept then click the description.

Concepts
Matches

Build, train, and deploy ML models

ETL and data cataloging

Object storage for datasets and models

Serverless compute for preprocessing

Image and video analysis

Why these pairings

The correct matches associate each service with its primary ML pipeline role. SageMaker is for end-to-end ML, Lambda for serverless inference, S3 for data/model storage, Glue for data preparation. Distractors confuse these roles.

Practice this question →
313
MCQeasy

A company is building a data pipeline to process streaming data from IoT devices. The data must be ingested with low latency, transformed in real-time using custom logic, and stored in Amazon S3 partitioned by device ID and timestamp. Which combination of AWS services should the company use to meet these requirements?

A.Amazon Kinesis Data Firehose with direct S3 delivery
B.Amazon Managed Streaming for Apache Kafka (MSK) with Amazon S3 sink connector
C.Amazon DynamoDB Streams with AWS Lambda and Amazon S3
D.Amazon Kinesis Data Streams with AWS Lambda and Amazon S3
AnswerD

Kinesis Data Streams for ingestion, Lambda for real-time transformation, and S3 for storage with partitioning.

Why this answer

Amazon Kinesis Data Streams provides low-latency ingestion of streaming data, AWS Lambda can apply custom transformation logic in real-time, and the transformed data can be stored in Amazon S3 with partitioning by device ID and timestamp using AWS Lambda to write to S3 with appropriate prefix. Option A is incorrect because Kinesis Data Firehose does not support custom transformation without invoking a Lambda function and cannot partition on write at the level of granularity required (device ID and timestamp). Option B is incorrect because Amazon MSK adds operational overhead and is more complex than needed; although an S3 sink connector can write to S3, it does not easily support custom transformation and partitioning by device ID and timestamp without additional configuration.

Option C is incorrect because DynamoDB Streams is designed for change data capture from DynamoDB tables and is not suitable for direct ingestion of high-volume IoT streaming data.

Practice this question →
314
MCQmedium

A data engineer needs to continuously ingest streaming data from thousands of IoT devices and store the raw data in Amazon S3 for archival processing. The data volume varies significantly throughout the day, and the solution must be serverless, scalable, and cost-effective. Which AWS service should be used to capture and buffer the streaming data before writing to S3?

A.Amazon Kinesis Data Firehose
B.Amazon Kinesis Data Streams
C.AWS Glue
D.Amazon Simple Queue Service (SQS)
AnswerA

Kinesis Data Firehose is a serverless service that can directly deliver streaming data to S3 with buffering.

Why this answer

Amazon Kinesis Data Firehose is the correct choice because it is a fully managed, serverless service designed to reliably capture, buffer, and automatically load streaming data into Amazon S3 without requiring any custom code or infrastructure management. It handles variable data volumes by scaling automatically and provides built-in buffering (up to 128 MB or 900 seconds) before writing to S3, making it cost-effective for archival storage.

Exam trap

The trap here is that candidates confuse Kinesis Data Streams (a real-time processing layer requiring custom consumers) with Kinesis Data Firehose (a managed delivery service), and overlook that Firehose's built-in buffering and direct S3 integration make it the serverless, cost-effective choice for archival ingestion.

How to eliminate wrong answers

Option B (Amazon Kinesis Data Streams) is wrong because it is a real-time data streaming service that requires consumers to process and write data to S3, and it does not provide built-in buffering or automatic S3 delivery; it is designed for custom real-time processing, not direct archival ingestion. Option C (AWS Glue) is wrong because it is a serverless ETL service for batch data transformation and cataloging, not a streaming ingestion or buffering service; it cannot capture or buffer streaming data in real time. Option D (Amazon Simple Queue Service - SQS) is wrong because it is a message queue for decoupling application components, not a streaming data ingestion service; it lacks the throughput, buffering, and automatic S3 delivery capabilities needed for high-volume IoT data.

Practice this question →
315
MCQhard

A data engineering team is designing a data pipeline to process large CSV files (10-50 GB each) stored in Amazon S3. The pipeline must transform the data using AWS Glue and load it into Amazon Redshift for analytics. The team wants to minimize costs while ensuring the pipeline can handle peak loads. Which approach is the most cost-effective?

A.Use AWS Lambda to process each file and load into Redshift.
B.Use Amazon EMR with Hive to transform the data and load into Redshift.
C.Use an AWS Glue Python shell job with a single r5.xlarge worker.
D.Use AWS Glue with Spark and dynamic frames, scaling the number of workers based on file size.
AnswerD

Correct: Glue Spark jobs handle large files efficiently; dynamic frames simplify schema handling.

Why this answer

AWS Glue with Spark and dynamic frames is the most cost-effective approach because it is serverless, automatically scales workers based on file size, and is optimized for ETL on large CSV files (10-50 GB) in S3. Dynamic frames provide built-in transformations and schema inference, reducing development effort, while the ability to adjust the number of workers allows handling peak loads without over-provisioning. This minimizes idle compute costs compared to always-on clusters like EMR.

Exam trap

The trap here is that candidates often choose AWS Lambda (Option A) for its low cost and simplicity, failing to recognize its strict execution limits (15-minute timeout, 10 GB memory) that make it impractical for multi-GB file processing, or they pick EMR (Option B) assuming it is always cheaper, ignoring the overhead of cluster management and idle costs.

How to eliminate wrong answers

Option A is wrong because AWS Lambda has a maximum execution timeout of 15 minutes and a 10 GB memory limit, making it unsuitable for processing 10-50 GB CSV files, and it lacks native support for complex transformations or direct Redshift loading at scale. Option B is wrong because Amazon EMR with Hive requires provisioning and managing a persistent cluster, incurring costs even when idle, and Hive is less performant for large-scale CSV transformations compared to Spark-based Glue jobs. Option C is wrong because an AWS Glue Python shell job runs on a single worker (r5.xlarge) with limited memory and no distributed processing, making it unable to handle 10-50 GB files efficiently, leading to out-of-memory errors or excessive runtime.

Practice this question →
316
MCQhard

A company runs a real-time recommendation system that uses Amazon SageMaker endpoints for inference. The system ingests user activity data from a mobile app via Amazon API Gateway and AWS Lambda, which writes events to an Amazon Kinesis Data Stream. A second Lambda function consumes the stream, calls a SageMaker endpoint to generate recommendations, and stores the results in Amazon DynamoDB. The system has been working well, but recently the team noticed an increase in latency from the time a user action occurs to when the recommendation is stored. The SageMaker endpoint shows increased invocation latency but no throttling. CloudWatch metrics show that the Kinesis stream's IteratorAgeMilliseconds is increasing, indicating the consumer is falling behind. The Lambda consumer's duration is within limits, but the number of invocations is lower than expected. The team suspects the issue is with the event source mapping. Which course of action should the team take to reduce the latency?

A.Increase the batch size in the event source mapping to process more records per invocation.
B.Increase the number of shards in the Kinesis data stream to increase parallelism.
C.Decrease the Lambda function's reserved concurrency to force it to scale down.
D.Replace the Lambda consumer with an Amazon Kinesis Data Firehose delivery stream.
AnswerA

Larger batches improve throughput by reducing overhead per invocation.

Why this answer

The increasing IteratorAgeMilliseconds indicates the consumer is falling behind. The Lambda consumer's duration is within limits but the number of invocations is lower than expected, suggesting that the event source mapping is not invoking the function often enough. Increasing the batch size allows each invocation to process more records per invocation, effectively increasing throughput without requiring more invocations.

This directly addresses the lag. Option B (increase shards) could help if the consumer had sufficient concurrency, but the root cause is low invocations per shard, not lack of shards. Option C (decrease reserved concurrency) would worsen the problem.

Option D (Firehose) does not solve the consumer lag and changes the architecture unnecessarily.

Practice this question →
317
Multi-Selectmedium

A company is building a data lake on Amazon S3 and wants to ensure that data is encrypted at rest using AWS KMS. Which TWO actions are required to achieve this? (Choose TWO.)

Select 2 answers
A.Configure the KMS key policy to allow the S3 service to use the key
B.Enable default encryption on the S3 bucket with SSE-KMS
C.Add a bucket policy that denies PutObject without encryption
D.Enable encryption in transit using HTTPS for all S3 API calls
E.Use client-side encryption on all data before uploading
AnswersA, B

The key policy must grant the S3 service principal permission to encrypt/decrypt.

Why this answer

AWS KMS uses key policies to control access to the KMS key. For S3 to use a KMS key for server-side encryption (SSE-KMS), the key policy must grant the S3 service principal (or the bucket owner's account) the necessary permissions, such as kms:Encrypt and kms:Decrypt. Without this policy, S3 cannot access the key to encrypt or decrypt objects at rest.

Option B is correct because enabling default encryption on the S3 bucket with SSE-KMS ensures that all objects uploaded to the bucket are automatically encrypted using the specified KMS key, meeting the requirement for encryption at rest.

Exam trap

The trap here is that candidates often confuse 'encryption at rest' with 'encryption in transit' or 'enforcing encryption via bucket policies,' and may select options like C or D, which address different security controls, instead of focusing on the specific mechanism (SSE-KMS) and the necessary KMS key policy configuration.

Practice this question →
318
Drag & Dropmedium

Drag and drop the steps to set up cross-validation in a SageMaker training job using the built-in XGBoost algorithm in the correct order.

Drag steps to the numbered slots on the right, or tap a step then tap a slot.

Steps
Order
1Step 1
2Step 2
3Step 3
4Step 4

Why this order

Cross-validation requires data splitting, job configuration with CV parameters, execution, and model selection.

Practice this question →
319
MCQeasy

A data scientist needs to run a one-time query on 10 TB of data stored in S3 using Amazon Athena. The query scans 5 TB and returns a small result set. Which approach minimizes cost?

A.Query the data directly in Athena without any preprocessing
B.Create an S3 Select query to filter data before Athena
C.Use Amazon Redshift Spectrum to query the data
D.Use AWS Glue to convert the data to Parquet format and repartition by date
AnswerA

For a one-time query, scanning 5 TB at $5 per TB is $25, which is minimal compared to preprocessing costs.

Why this answer

Athena charges based on the amount of data scanned per query. Since this is a one-time query on 10 TB of data that scans only 5 TB, querying directly in Athena without preprocessing is the most cost-effective approach because you pay only for the 5 TB scanned, with no additional costs for data conversion, storage, or cluster provisioning.

Exam trap

The trap here is that candidates assume data must be converted to a columnar format (like Parquet) to reduce costs, ignoring that for a one-time query, the cost of conversion and storage outweighs the savings from reduced scan size.

How to eliminate wrong answers

Option B is wrong because S3 Select is designed for filtering data within a single object (e.g., a CSV or JSON file) and cannot be used as a preprocessing step before Athena; it operates at the object level, not across multiple objects or as a query pipeline. Option C is wrong because Redshift Spectrum requires provisioning an Amazon Redshift cluster (even if serverless, it incurs compute costs) and is overkill for a one-time query, leading to higher costs than Athena's pay-per-scan model. Option D is wrong because converting the data to Parquet and repartitioning by date using AWS Glue would incur significant costs for the ETL job and storage, and is unnecessary for a one-time query where the cost of scanning 5 TB directly in Athena is lower than the combined conversion and storage costs.

Practice this question →
320
MCQmedium

A data engineer is responsible for managing a data lake on Amazon S3. The data lake contains CSV files from various sources, totaling 10 TB. The engineer needs to make this data queryable using Amazon Athena. However, Athena queries are currently taking a long time and scanning large amounts of data. The engineer has noticed that the CSV files are not partitioned, and there are no indexes. The engineer wants to improve query performance and reduce costs. The data is accessed frequently for the last 30 days, but older data is rarely queried. The engineer also wants to minimize the amount of data scanned by Athena. What should the engineer do?

A.Convert the CSV files to JSON format and use Athena to query them.
B.Convert the CSV files to Parquet format and partition the data by date.
C.Create indexes on the S3 objects using AWS Glue.
D.Convert the CSV files to ORC format and create a view in Athena.
AnswerB

Parquet is columnar and compressed; partitioning by date allows partition pruning, reducing scan size.

Why this answer

The best choice. Converting CSV to Parquet reduces data scanned due to columnar storage and compression. Partitioning by date allows Athena to skip older data that is rarely queried, further minimizing scan size and cost.

Option A (JSON) does not improve performance significantly and still lacks partitioning. Option C is invalid because Athena does not support indexes. Option D (ORC) is columnar but without partitioning it performs worse than Parquet with partitioning, and views do not reduce scan size.

Practice this question →
321
MCQeasy

A company uses Amazon RDS for its transactional database and needs to export a daily snapshot of a table to Amazon S3 in Parquet format for analytics. Which AWS service can perform this export without writing custom code?

A.Amazon Redshift
B.AWS Database Migration Service (DMS)
C.Amazon Athena
D.AWS Glue
AnswerD

Glue can run scheduled ETL jobs to extract from RDS and write to S3 in Parquet.

Why this answer

AWS Glue is correct because it provides a fully managed ETL service that can natively read from Amazon RDS and write to Amazon S3 in Parquet format using a scheduled job, without requiring any custom code. Glue's built-in transform capabilities and crawlers can convert the data to columnar Parquet format efficiently for analytics workloads.

Exam trap

The trap here is that candidates often confuse AWS Glue's ETL capabilities with AWS DMS's migration focus, assuming DMS can handle scheduled format conversions like Parquet, but DMS primarily deals with ongoing replication and does not natively support Parquet output without custom transformation.

How to eliminate wrong answers

Option A is wrong because Amazon Redshift is a data warehouse service for querying structured data, not a tool for exporting data from RDS to S3 in Parquet format; it can load data from S3 but does not perform scheduled exports from RDS to S3. Option B is wrong because AWS DMS is designed for continuous database migration and replication, not for scheduled daily snapshots to S3 in Parquet format; while DMS can write to S3, it requires custom transformation tasks and does not natively output Parquet without additional configuration. Option C is wrong because Amazon Athena is an interactive query service for analyzing data in S3 using SQL, not a service for exporting or transforming data from RDS to S3; it cannot initiate data movement from RDS to S3.

Practice this question →
322
Multi-Selectmedium

A data engineer is building a streaming pipeline using Amazon Kinesis Data Streams and AWS Lambda. The Lambda function processes records and writes results to Amazon S3. The engineer notices that the Lambda function is experiencing throttling and some records are being dropped. Which TWO actions should the engineer take to improve the reliability of the pipeline?

Select 2 answers
A.Increase the number of shards in the Kinesis data stream.
B.Set a reserved concurrency on the Lambda function to prevent other functions from using its capacity.
C.Add a Dead Letter Queue to the Lambda function to capture failed records.
D.Decrease the batch size in the Lambda event source mapping.
E.Increase the Kinesis stream's retention period to 7 days.
AnswersA, B

More shards increase parallelism and throughput.

Why this answer

Increasing the number of shards in the Kinesis data stream directly increases the stream's throughput capacity. Each shard supports up to 1 MB/s write and 2 MB/s read, so more shards allow the stream to handle higher data volumes, reducing the likelihood of throttling and dropped records at the stream level.

Exam trap

The trap here is that candidates often confuse stream-level throttling with Lambda processing failures, leading them to choose a Dead Letter Queue (which handles processing failures) instead of addressing the root cause of insufficient throughput or concurrency.

Practice this question →
323
MCQeasy

A data engineer wants to stream clickstream data from a web application to Amazon S3 for near-real-time analytics. Which AWS service should be used to ingest and buffer the data before landing in S3?

A.Amazon AppFlow
B.Amazon Kinesis Data Streams
C.Amazon Kinesis Data Firehose
D.AWS Glue
AnswerC

Firehose can directly deliver streaming data to S3.

Why this answer

Amazon Kinesis Data Firehose is the correct choice because it is a fully managed service designed to ingest streaming data, buffer it, and reliably deliver it to destinations like Amazon S3 with near-real-time latency (typically 60 seconds). It handles automatic scaling, data transformation, and compression, making it ideal for clickstream data landing directly into S3 for analytics.

Exam trap

The trap here is that candidates confuse Amazon Kinesis Data Streams with Kinesis Data Firehose, not realizing that Data Streams requires a separate consumer to write to S3, while Firehose is purpose-built for direct, buffered delivery to S3.

How to eliminate wrong answers

Option A is wrong because Amazon AppFlow is a managed integration service for transferring data between SaaS applications (e.g., Salesforce, Slack) and AWS, not for streaming clickstream data from a web application. Option B is wrong because Amazon Kinesis Data Streams is a real-time data streaming service that requires custom consumers and does not natively buffer or deliver data to S3; it is intended for custom stream processing, not direct S3 ingestion. Option D is wrong because AWS Glue is a serverless ETL service for batch data preparation and cataloging, not a streaming ingestion or buffering service.

Practice this question →
324
MCQmedium

A company uses AWS Glue jobs with job bookmarks enabled to process incremental data. They notice that the job processes all data each time instead of only new data. What is the most likely reason?

A.The TempDir is not configured correctly.
B.The job bookmark option is set to 'job-bookmark-enable' but should be 'job-bookmark-disable'.
C.The source data does not have a column that can be used as a bookmark key.
D.The MaxConcurrentRuns is set to 3, which can cause bookmark conflicts.
AnswerD

Multiple concurrent runs can corrupt bookmark state.

Why this answer

Setting MaxConcurrentRuns to a value greater than 1 can cause job bookmark conflicts. When multiple concurrent runs of the same AWS Glue job attempt to update the bookmark state simultaneously, they can overwrite each other's progress, leading to inconsistent bookmark tracking. This results in the job reprocessing all data instead of only incremental data, as the bookmark fails to correctly record the last processed position.

Exam trap

The trap here is that candidates often overlook the impact of concurrent runs on bookmark state, assuming that parallelism only affects performance, not data integrity, and instead focus on superficial configuration issues like TempDir or bookmark key columns.

How to eliminate wrong answers

Option A is wrong because TempDir is used for temporary staging of data (e.g., for Spark shuffle operations or schema evolution), not for bookmark functionality; an incorrect TempDir would cause runtime errors, not reprocessing all data. Option B is wrong because 'job-bookmark-enable' is the correct parameter value to enable job bookmarks; setting it to 'job-bookmark-disable' would disable bookmarks entirely, which would cause full reprocessing, but the question states the job is intended to process incremental data, so this option misrepresents the correct configuration. Option C is wrong because AWS Glue job bookmarks do not require a specific column as a bookmark key; they use internal tracking mechanisms (e.g., file modification timestamps for S3, or primary key ordering for JDBC) to determine new data, and the absence of a suitable column would cause bookmarks to fail silently or process all data, but this is not the most likely reason given the symptom of reprocessing all data each time.

Practice this question →
325
MCQmedium

Refer to the exhibit. An IAM policy is attached to a data engineering team's role. The team needs to upload data to the 'confidential' prefix in the 'my-data-lake' bucket. However, they are receiving 'AccessDenied' errors. What is the likely cause?

A.The condition in the Deny statement requires the team to use a specific source IP address.
B.The Allow statement only grants GetObject and PutObject, but the team needs ListBucket.
C.The Deny statement with the condition explicitly denies access to the 'confidential' prefix for accounts other than 123456789012.
D.The Allow statement's resource does not include the 'confidential' prefix.
AnswerC

The Deny statement applies to all actions on the confidential prefix for accounts not matching 123456789012, overriding the Allow.

Why this answer

The Deny statement explicitly denies access to the 'confidential' prefix when the request comes from an AWS account other than 123456789012. Since the data engineering team's role likely belongs to a different account, the Deny condition matches and overrides any Allow statements, resulting in an 'AccessDenied' error. In IAM, an explicit Deny always takes precedence over an Allow, so even if the Allow statement grants PutObject, the Deny blocks the upload.

Exam trap

The AWS exam often tests the principle that an explicit Deny overrides any Allow, and candidates mistakenly focus on the Allow statement's permissions (like missing ListBucket) instead of recognizing that the Deny statement with a condition is the root cause of the 'AccessDenied' error.

How to eliminate wrong answers

Option A is wrong because the condition in the Deny statement uses 'StringNotEquals' with 'aws:SourceIp', which denies access if the source IP is not a specific value, but the exhibit shows the condition is on 'aws:SourceAccount', not source IP. Option B is wrong because the team is receiving 'AccessDenied' when uploading (PutObject), and ListBucket is not required for uploading objects; the error is not due to missing ListBucket permissions. Option D is wrong because the Allow statement's resource 'arn:aws:s3:::my-data-lake/confidential/*' does include the 'confidential' prefix, so the Allow is correctly scoped; the issue is the overriding Deny.

Practice this question →
326
MCQeasy

An S3 event notification is configured to trigger a Lambda function when new objects are created. The Lambda function processes the event JSON shown. Which field should the function use to read the new object from S3?

A.s3.s3SchemaVersion
B.awsRegion
C.eventName
D.s3.bucket.arn and s3.object.key
AnswerD

These provide the bucket ARN and object key.

Why this answer

The Lambda function needs the bucket name and object key to retrieve the new object from S3. The `s3.bucket.arn` provides the bucket identifier, and `s3.object.key` provides the object path. Together, they allow the function to call `GetObject` on the S3 API.

Exam trap

The trap here is that candidates confuse metadata fields like `eventName` or `awsRegion` with the actual object location, overlooking that only the bucket ARN and object key together provide the necessary S3 coordinates for retrieval.

How to eliminate wrong answers

Option A is wrong because `s3.s3SchemaVersion` indicates the version of the S3 event notification schema, not the object location. Option B is wrong because `awsRegion` specifies the AWS region of the bucket, which is not sufficient to identify or read a specific object. Option C is wrong because `eventName` describes the type of S3 event (e.g., 'ObjectCreated:Put'), not the object identifier.

Practice this question →
327
MCQmedium

A company uses Kinesis Data Streams to ingest real-time sensor data. The data is consumed by a Lambda function that writes to DynamoDB. During peak hours, the Lambda function throws ProvisionedThroughputExceededException. The team wants to decouple the write operation and improve resilience. What should they do?

A.Use Kinesis Firehose as a consumer of the stream, with a Lambda transformation to write to DynamoDB, and enable error handling.
B.Increase the Lambda function's reserved concurrency and provision more DynamoDB write capacity.
C.Place the Lambda function's output into an Amazon SQS queue, and have a second Lambda function write to DynamoDB.
D.Use Kinesis Data Analytics to process the stream and write results directly to DynamoDB.
AnswerA

Firehose buffers data, retries on failures, and decouples the producer from DynamoDB writes.

Why this answer

Kinesis Firehose can consume data from a Kinesis Data Stream and invoke a Lambda function for transformation before delivering to destinations like DynamoDB. By using Firehose with error handling, the team decouples the write operation from the Lambda consumer, allowing Firehose to buffer data and retry failed writes, which improves resilience against ProvisionedThroughputExceededException without losing data.

Exam trap

The trap here is that candidates often assume adding a queue (SQS) is the standard decoupling pattern, but in this context, Kinesis Firehose is purpose-built for stream ingestion with built-in error handling and Lambda integration, making it a more direct and efficient solution than introducing an additional queue layer.

How to eliminate wrong answers

Option B is wrong because increasing Lambda reserved concurrency and DynamoDB write capacity only scales the existing tightly coupled architecture, not decoupling it; it does not address the root cause of throttling during peak hours and may lead to higher costs without resilience. Option C is wrong because placing Lambda output into an SQS queue and having a second Lambda write to DynamoDB adds unnecessary complexity and latency, and SQS does not natively integrate with DynamoDB for batch writes; it also fails to leverage the existing Kinesis stream's ordered processing. Option D is wrong because Kinesis Data Analytics is designed for real-time analytics using SQL or Apache Flink, not for direct writes to DynamoDB; it cannot write results to DynamoDB natively and would require additional downstream processing, making it an inappropriate decoupling solution.

Practice this question →
328
MCQeasy

A company uses Amazon Kinesis Data Firehose to deliver data to an Amazon S3 bucket. The data is organized by year/month/day/hour. The team needs to ensure that all data is encrypted at rest in S3 using an AWS KMS customer managed key (CMK). Which configuration should the team implement?

A.Configure the S3 bucket's default encryption to use the customer managed KMS key.
B.Use an AWS Lambda function to encrypt the data after it is delivered to S3.
C.In the Firehose delivery stream configuration, enable S3 destination encryption and select the customer managed KMS key.
D.Add a bucket policy that denies PutObject unless the request includes the correct KMS key.
AnswerC

Firehose supports SSE-KMS for the S3 destination directly.

Why this answer

The correct approach. Kinesis Data Firehose can be configured to encrypt data at rest in S3 using AWS KMS. In the Firehose delivery stream configuration, under S3 destination settings, you can enable encryption and select a customer managed KMS key.

This ensures all data written to S3 is encrypted with that key. Option A is incorrect because S3 default encryption applies to objects uploaded directly to S3, but Firehose writes objects using its own IAM role and can override default encryption; configuring encryption in Firehose is the recommended way. Option B is incorrect because using a Lambda function to encrypt after delivery adds unnecessary complexity and latency; Firehose can encrypt natively.

Option D is incorrect because a bucket policy denying PutObject without the correct KMS key would work but is not the simplest or most straightforward configuration; Firehose can handle encryption directly.

Practice this question →
329
Multi-Selecteasy

A company needs to move 50 TB of data from an on-premises data center to Amazon S3. The company has a limited internet bandwidth of 100 Mbps. The data transfer must be completed within 10 days. Which TWO services should the company use together to meet these requirements?

Select 2 answers
A.Amazon S3 as the destination
B.AWS Direct Connect
C.AWS Site-to-Site VPN
D.AWS Snowball Edge
E.AWS DataSync over the internet
AnswersA, D

Data is ultimately stored in S3.

Why this answer

The correct answers are A (Amazon S3 as the destination) and D (AWS Snowball Edge). The company needs to transfer 50 TB within 10 days, but with only 100 Mbps bandwidth, only about 10.8 TB can be transferred over the internet in that time. Therefore, a physical transfer solution like AWS Snowball Edge is required to move the data offline to S3.

Option B (AWS Direct Connect) is incorrect because even though it provides a dedicated connection, it still requires time to provision and would not be able to transfer 50 TB within 10 days if the bandwidth is limited to 100 Mbps (or even higher, Snowball is more practical for such large data). Option C (AWS Site-to-Site VPN) is incorrect because it also uses the internet and is subject to the same bandwidth limitations. Option E (AWS DataSync over the internet) is incorrect because it is also limited by the 100 Mbps bandwidth and cannot complete the transfer within 10 days.

Practice this question →
330
MCQhard

A company is using Amazon Redshift for data warehousing. The data engineering team notices that queries are slow and the system is frequently writing to disk due to insufficient memory. Which type of workload management (WLM) configuration change would help reduce disk writes?

A.Increase the number of query concurrency slots.
B.Increase the memory percentage allocated to the WLM queue.
C.Enable query monitoring rules to abort queries that spill to disk.
D.Enable short query acceleration (SQA).
AnswerB

More memory per query reduces disk spill.

Why this answer

When queries spill to disk in Amazon Redshift, it indicates that the memory allocated to the WLM queue is insufficient for the workload. Increasing the memory percentage for the queue allows more queries to be processed in memory, reducing the need to write intermediate results to disk and improving query performance.

Exam trap

The trap here is that candidates often confuse increasing concurrency (Option A) with improving performance, not realizing that higher concurrency reduces per-query memory and increases disk spills, making the problem worse.

How to eliminate wrong answers

Option A is wrong because increasing the number of query concurrency slots actually reduces the memory available per slot, which can increase disk spills and worsen performance. Option C is wrong because query monitoring rules that abort queries spilling to disk do not reduce disk writes; they simply terminate the queries, which is a reactive measure and does not address the underlying memory shortage. Option D is wrong because short query acceleration (SQA) prioritizes short-running queries but does not increase memory allocation or directly reduce disk spills for memory-intensive queries.

Practice this question →
331
MCQmedium

A data engineer needs to design a data pipeline that ingests CSV files from an SFTP server daily, transforms them, and loads them into Amazon Redshift. The files are typically 2-3 GB. Which combination of AWS services is MOST appropriate?

A.Use AWS Glue ETL with a JDBC connection to the SFTP server to read files directly.
B.Use AWS Lambda to download the files from SFTP, transform them in memory, and write to Redshift using the Data API.
C.Use AWS Transfer Family to automate SFTP file retrieval to S3, then use Redshift COPY to load data.
D.Use Amazon Kinesis Data Firehose with an HTTP endpoint source to receive files from SFTP.
AnswerC

Transfer Family handles SFTP natively, and COPY loads data efficiently into Redshift.

Why this answer

The most appropriate because AWS Transfer Family provides a fully managed, serverless solution for automating SFTP file retrieval directly into Amazon S3. Once the CSV files are in S3, the Redshift COPY command can efficiently load the 2-3 GB files using parallel processing, which is far more performant and cost-effective than alternatives like Lambda or Glue for large file sizes.

Exam trap

The trap here is that candidates may assume AWS Glue or Lambda can handle SFTP directly, but they lack native SFTP support and are not designed for large file transfers, while AWS Transfer Family is purpose-built for this exact use case.

How to eliminate wrong answers

Option A is wrong because AWS Glue ETL with a JDBC connection to an SFTP server is not a supported pattern; JDBC is for relational databases, not file transfer protocols like SFTP (which uses SSH). Option B is wrong because AWS Lambda has a maximum execution timeout of 15 minutes and a 10 GB memory limit, making it unsuitable for transforming 2-3 GB files in memory, and the Redshift Data API is designed for small queries, not bulk data loading. Option D is wrong because Amazon Kinesis Data Firehose with an HTTP endpoint source expects streaming data via HTTP POST, not batch file retrieval from an SFTP server, and it cannot natively connect to SFTP.

Practice this question →
332
MCQhard

An e-commerce company uses Amazon DynamoDB as the primary data store for user sessions. They want to run analytics on historical session data using Amazon Athena. What is the recommended approach to export DynamoDB data to S3 in a format optimized for Athena?

A.Use AWS Data Pipeline to copy data to S3 as CSV
B.Use Amazon Kinesis Data Firehose to stream data from DynamoDB to S3
C.Use DynamoDB Streams with AWS Lambda to write to S3 as JSON
D.Use AWS Glue ETL to read from DynamoDB and write to S3 as Parquet
AnswerD

Glue can efficiently export data and convert to columnar format.

Why this answer

AWS Glue ETL can read from DynamoDB and write to S3 in Parquet format, which is optimized for Athena due to its columnar storage and compression. Option A (AWS Data Pipeline) can copy data to S3 as CSV, but CSV is less efficient for Athena and Data Pipeline is a legacy service. Option B (Amazon Kinesis Data Firehose) is designed for streaming data, not for exporting existing DynamoDB tables.

Option C (DynamoDB Streams with Lambda) writes to S3 as JSON, which is less performant than Parquet for Athena queries and adds operational complexity.

Practice this question →
333
MCQhard

Refer to the exhibit. A data engineer has attached this IAM policy to an IAM role used by an AWS Glue ETL job. The job reads from an S3 bucket (data-bucket) that is encrypted with SSE-KMS using the key arn:aws:kms:us-east-1:123456789012:key/abc123, transforms the data, and writes the result to a different S3 bucket (output-bucket) encrypted with a different KMS key (arn:aws:kms:us-east-1:123456789012:key/xyz789). When the job runs, it fails with an access denied error. What is the cause?

A.The policy does not include s3:GetObject permission for the output bucket.
B.The policy does not include glue:CreateTable permission.
C.The policy does not include s3:PutObject permission for the output bucket.
D.The policy does not grant kms:Encrypt permission for the output bucket's KMS key.
AnswerD

To write to an SSE-KMS encrypted bucket, the role needs kms:Encrypt or kms:GenerateDataKey for that key.

Why this answer

The job fails because the IAM policy grants kms:Decrypt and kms:GenerateDataKey for the input bucket's KMS key (abc123) but does not grant kms:Encrypt or kms:GenerateDataKey for the output bucket's KMS key (xyz789). To write encrypted data to the output bucket, the AWS Glue job must have permission to encrypt using the output KMS key. Option D is correct because the missing kms:Encrypt permission causes the access denied error.

Option A is incorrect because the policy includes s3:GetObject for the input bucket. Option B is incorrect because Glue catalog permissions are not relevant to the encryption error. Option C is incorrect because the error is due to missing KMS permissions for the output bucket, not because the policy does not include s3:PutObject.

Practice this question →
334
MCQmedium

A data engineering team needs to move 10 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The data is currently stored in HDFS. Which service should they use for an efficient transfer?

A.AWS DataSync
B.S3 Transfer Acceleration
C.Amazon Kinesis Data Streams
D.AWS Snowball Edge
AnswerA

DataSync can transfer data from HDFS to S3.

Why this answer

AWS DataSync is the correct choice because it is designed to efficiently transfer large volumes of data from on-premises storage systems, including HDFS, to AWS services like Amazon S3. It uses a purpose-built network protocol and parallel multi-threading to optimize transfer speed over the internet or AWS Direct Connect, and it can handle the 10 TB volume without requiring physical appliances or complex streaming setups.

Exam trap

Candidates often mistakenly choose S3 Transfer Acceleration thinking it is for general data migration, but it only speeds up uploads to S3 and lacks HDFS integration and management features provided by AWS DataSync.

How to eliminate wrong answers

Option B is wrong because S3 Transfer Acceleration only speeds up uploads to S3 over the internet by using AWS edge locations, but it does not integrate with or understand HDFS, so it cannot directly read data from an on-premises Hadoop cluster. Option C is wrong because Amazon Kinesis Data Streams is a real-time streaming service for ingesting small records (up to 1 MB per record) and is not designed for batch transfer of 10 TB of historical data from HDFS. Option D is wrong because AWS Snowball Edge is a physical device for offline data transfer, which is unnecessary when the network bandwidth is sufficient for a 10 TB transfer; DataSync is more efficient for online transfers.

Practice this question →
335
Drag & Dropmedium

Drag and drop the steps to perform hyperparameter tuning using SageMaker Automatic Model Tuning in the correct order.

Drag steps to the numbered slots on the right, or tap a step then tap a slot.

Steps
Order
1Step 1
2Step 2
3Step 3
4Step 4

Why this order

Tuning involves defining search space, creating a tuning job, setting limits, executing, and selecting best model.

Practice this question →
336
MCQeasy

A data engineer is building a data pipeline to process user clickstream data. The data arrives as JSON files in an S3 bucket. The pipeline must transform the JSON into Parquet format and partition by date and event type, then make the data available for Amazon Athena queries. The engineer needs a fully managed, serverless solution with minimal operational overhead. Which combination of AWS services should the engineer use?

A.Use Amazon EMR with Spark to read JSON, convert to Parquet, and partition, then query with Athena.
B.Use AWS Glue ETL jobs to read JSON from S3, transform to Parquet, and write to a partitioned S3 location, then use Athena.
C.Use S3 Event Notifications to trigger an AWS Lambda function that converts the JSON to Parquet and writes to a partitioned S3 location, then query with Athena.
D.Use Amazon Kinesis Firehose to ingest data and convert to Parquet, then write to S3, and query with Athena.
AnswerC

Lambda is serverless, cost-effective for per-file processing, and can partition output easily.

Why this answer

AWS Lambda triggered by S3 Event Notifications provides a fully serverless, event-driven architecture with minimal operational overhead for converting JSON to Parquet and partitioning by date and event type. Lambda can process each new JSON file as it arrives, perform the transformation in memory (using libraries like PyArrow or Pandas), and write the Parquet output to a partitioned S3 path, which Athena can then query directly. This approach avoids managing any clusters or job scheduling, aligning with the requirement for a fully managed, serverless solution.

Exam trap

The MLS-C01 exam often tests the misconception that AWS Glue is the only serverless ETL option, but the trap here is that Lambda with S3 Event Notifications is a simpler, fully serverless alternative for file-based transformations when the workload fits within Lambda's constraints.

How to eliminate wrong answers

Option A is wrong because Amazon EMR with Spark requires provisioning and managing a cluster (even if ephemeral), incurring operational overhead and not being fully serverless; it also introduces complexity for a simple transformation task. Option B is wrong because AWS Glue ETL jobs, while serverless, involve job scheduling, startup latency, and cost for each job run, and are overkill for a real-time, event-driven pipeline where Lambda can handle the transformation more efficiently with lower latency and cost. Option D is wrong because Amazon Kinesis Firehose is designed for streaming data ingestion, not for batch processing of existing JSON files in S3; it cannot be triggered by S3 events to process files already stored, and its Parquet conversion is limited to the Firehose delivery stream, not arbitrary file transformations.

Practice this question →
337
Multi-Selectmedium

A data engineering team is designing a data pipeline that processes streaming data from Amazon Kinesis Data Streams using AWS Lambda. The team notices that some records are being processed multiple times (duplicates). Which TWO steps should the team take to ensure exactly-once processing?

Select 2 answers
A.Design the Lambda function to be idempotent.
B.Use a unique record identifier and store processed IDs in an external store like DynamoDB.
C.Increase the batch size to reduce the number of invocations.
D.Use Kinesis Producer Library (KPL) to guarantee exactly-once delivery.
E.Disable retries on the Lambda function.
AnswersA, B

Idempotency ensures repeated processing produces same result.

Why this answer

Options A and B are correct. Making the Lambda function idempotent ensures that processing the same record multiple times does not cause duplicates downstream. Using a unique identifier per record and storing processed IDs in an external store like DynamoDB allows deduplication by checking if a record has already been processed.

Option C is incorrect because increasing batch size does not prevent duplicates and may increase the chance of processing failures. Option D is incorrect because KPL provides exactly-once delivery to Kinesis Data Streams, not from the stream to Lambda, so deduplication is still needed. Option E is incorrect because disabling retries can lead to data loss without guaranteeing exactly-once processing.

Practice this question →
338
MCQmedium

A company is building a data lake on Amazon S3 and wants to use AWS Glue to catalog the data. The data includes CSV, Parquet, and JSON files. The team wants to ensure that the Glue crawler can infer the schema correctly and update the Data Catalog when new partitions are added. Which crawler configuration should be used?

A.Create separate crawlers for each file format and schedule them at different times.
B.Use a crawler that only catalogs Parquet files because they are more efficient.
C.Use a crawler with 'Update all new and existing partitions' disabled to avoid schema conflicts.
D.Create a single crawler that includes all file extensions and set the 'Update all new and existing partitions' option.
AnswerD

Correct: Single crawler with partition updates ensures comprehensive cataloging.

Why this answer

A single AWS Glue crawler can handle multiple file formats (CSV, Parquet, JSON) in a data lake on Amazon S3, and enabling 'Update all new and existing partitions' ensures the Data Catalog is refreshed with both new partitions and any schema changes in existing partitions. This configuration maintains a consistent and up-to-date catalog without manual intervention, which is essential for downstream analytics and machine learning workloads.

Exam trap

The trap here is that candidates mistakenly think disabling partition updates prevents schema conflicts, but in reality, it causes stale metadata for existing partitions, while a single crawler with updates enabled correctly handles schema evolution across all file formats.

How to eliminate wrong answers

Option A is wrong because creating separate crawlers for each file format introduces unnecessary complexity and overhead; a single crawler can efficiently catalog multiple formats, and scheduling them at different times may cause catalog inconsistencies. Option B is wrong because restricting the crawler to only Parquet files ignores CSV and JSON data, leading to incomplete cataloging and missing data for downstream processing. Option C is wrong because disabling 'Update all new and existing partitions' prevents the crawler from detecting schema changes in existing partitions, which can result in stale or incorrect metadata in the Data Catalog.

Practice this question →
339
MCQeasy

A company uses Amazon Kinesis Data Streams to ingest clickstream data from a website. The data is consumed by a custom application that runs on Amazon EC2 instances. The company notices that the consumer application is falling behind the producer, causing data to be throttled. Which action should the company take to improve the consumer's throughput?

A.Reduce the data retention period of the stream
B.Increase the number of shards in the Kinesis data stream
C.Increase the maximum concurrency of the AWS Lambda function that processes the stream
D.Use Amazon Kinesis Data Firehose to deliver data to Amazon S3
AnswerB

More shards increase the stream's read and write capacity.

Why this answer

Increasing the number of shards increases the stream's read capacity, allowing more consumers to read in parallel and improving throughput. Option A is wrong because reducing the data retention period does not increase read throughput; it only affects how long data is stored. Option C is wrong because Lambda concurrency is applicable only to Lambda functions, not to the custom EC2 application consuming the stream.

Option D is wrong because Amazon Kinesis Data Firehose is a different service for delivering streaming data to destinations like S3, and it does not improve the throughput of the existing EC2 consumer.

Practice this question →
340
MCQhard

A company runs an e-commerce platform that generates clickstream data in real-time. The data is ingested into Amazon Kinesis Data Streams (100 shards) and processed by AWS Lambda functions, which aggregate data in 1-minute windows and write the results to Amazon S3. The Lambda functions are triggered by the Kinesis stream using the event source mapping. Recently, the company noticed that some records are being processed multiple times, leading to duplicate data in S3. The Lambda function is idempotent, but the duplicates are causing downstream issues. The Lambda function's concurrency limit is 1000, and the batch size is 100. The average processing time per record is 200 ms. What is the most likely cause of the duplicates, and how should it be fixed?

A.Increase the Lambda concurrency limit to 2000 to handle the load.
B.Ensure the Lambda function is idempotent and uses the sequence number to deduplicate records.
C.Decrease the batch size to 10 to reduce the impact of failures.
D.Use Amazon SQS FIFO queue as a buffer between Kinesis and Lambda to guarantee exactly-once processing.
AnswerB

If the function fails and retries, using sequence numbers allows it to skip already processed records, preventing duplicates.

Why this answer

Lambda functions process records from Kinesis in batches. If the function fails (e.g., due to timeout or error), the entire batch is retried, causing duplicates if some records were already partially processed. To avoid duplicates, the function should be idempotent and should not commit partial results.

Option A is wrong because the concurrency is sufficient. Option C is wrong because increasing batch size increases the risk of partial failure. Option D is wrong because a FIFO queue does not integrate with Kinesis.

Practice this question →
341
MCQhard

An organization is migrating its on-premises Hadoop cluster to AWS. The cluster runs Spark jobs that process 50 TB of data daily. The data is stored in HDFS with 3x replication. Which storage option on AWS provides the best price-performance for this workload?

A.Use AWS Glue to run Spark jobs with data stored in S3
B.Use Amazon EMR with S3 as the data store via EMRFS
C.Use Amazon Redshift Spectrum to query the data directly in S3
D.Use Amazon EMR with HDFS on EBS volumes
AnswerB

S3 provides 11 9's durability and is cheaper than EBS. EMRFS seamlessly integrates with Spark.

Why this answer

Amazon EMR with S3 as the data store via EMRFS provides the best price-performance for this workload because it eliminates the need for 3x replication (S3 is inherently durable and replicated across multiple AZs), reduces storage costs, and allows compute and storage to scale independently. EMRFS enables Spark jobs to read/write directly to S3 with consistency guarantees, matching the throughput requirements of 50 TB daily processing without the overhead of managing HDFS on EBS volumes.

Exam trap

The trap here is that candidates assume HDFS replication is necessary for durability on AWS, overlooking that S3 provides built-in replication and durability, making EMRFS with S3 the cost-effective and performant choice for Spark workloads.

How to eliminate wrong answers

Option A is wrong because AWS Glue is a serverless Spark runtime that is not optimized for processing 50 TB daily at the same price-performance as EMR; it lacks fine-grained tuning and incurs higher costs for large-scale, predictable workloads. Option C is wrong because Redshift Spectrum is designed for SQL-based querying of data in S3, not for running Spark jobs; it cannot execute Spark transformations or leverage the Spark execution engine. Option D is wrong because using HDFS on EBS volumes replicates the on-premises 3x replication model, incurring high storage costs and operational overhead without leveraging S3's durability or elasticity, leading to worse price-performance.

Practice this question →
342
MCQhard

A Glue job fails with an AccessDenied error when trying to write to the S3 bucket my-data-lake. The IAM policy attached to the job role is shown in the exhibit. What is the MOST likely reason for the failure?

A.The s3:ListBucket action is missing on the bucket level
B.The job role does not have permissions to decrypt the KMS key used for server-side encryption
C.The s3:PutObject action is not sufficient; the job needs s3:PutObjectAcl
D.The resource ARN for s3:PutObject should include a specific prefix
AnswerB

SSE-KMS requires kms:Decrypt and kms:GenerateDataKey permissions, which are missing.

Why this answer

The policy allows s3:PutObject on the bucket, so write access seems granted. However, if the bucket is encrypted with SSE-KMS, the job also needs kms:Decrypt and kms:GenerateDataKey permissions. The policy does not include KMS actions.

The bucket policy might also deny, but the most common issue is KMS encryption.

Practice this question →
343
MCQeasy

A team stores raw data in S3 and uses a Glue Data Catalog for metadata. They want to allow data scientists to query the data with Amazon Athena using their existing IAM roles. What is the MINIMUM set of permissions required?

A.Grant the IAM role permissions for Athena, Glue, and S3 (read and write).
B.Grant the IAM role permissions for Athena actions, Glue Data Catalog actions, and S3 read access.
C.Grant the IAM role permissions for Athena and Amazon Redshift Spectrum.
D.Grant the IAM role permissions for Athena and Amazon Kinesis.
AnswerB

Athena requires GetTable, GetDatabase, etc. from Glue, and GetObject from S3.

Why this answer

Athena requires read access to S3 for querying data, permissions to the Glue Data Catalog for schema/metadata resolution, and Athena-specific actions to run queries. Write access to S3 is not required for querying, making B the minimum set.

Exam trap

The trap here is that candidates often assume Athena requires full S3 read/write access, but only read is needed for querying; write is only needed if the query outputs results to S3, which is not part of the minimum set for querying.

How to eliminate wrong answers

Option A is wrong because it includes S3 write access, which is unnecessary for querying and violates the principle of least privilege. Option C is wrong because Amazon Redshift Spectrum is a separate service for querying data in Redshift, not Athena, and is not required for Athena queries. Option D is wrong because Amazon Kinesis is a streaming data service unrelated to Athena query execution.

Practice this question →
344
MCQeasy

A data engineer needs to transfer 50 TB of historical data from an on-premises Hadoop cluster to Amazon S3. The company has a 100 Mbps internet connection and a tight deadline of two weeks. Which AWS service should the engineer use to transfer the data most efficiently?

A.AWS Storage Gateway (Volume Gateway)
B.AWS Snowball Edge
C.Amazon S3 Transfer Acceleration
D.AWS DataSync over the internet
AnswerB

Snowball Edge provides physical shipping, bypassing bandwidth limitations.

Why this answer

B is correct because transferring 50 TB over a 100 Mbps connection would take approximately 48 days (50 TB * 8 / 100 Mbps / 86400 seconds/day), far exceeding the two-week deadline. AWS Snowball Edge is a physical data transport device that can securely transfer petabytes of data offline, bypassing network bandwidth constraints entirely. For large datasets and tight deadlines, Snowball Edge is the most efficient AWS service.

Exam trap

The trap here is that candidates may overestimate the effectiveness of network optimization services like S3 Transfer Acceleration or DataSync, failing to calculate that even with perfect efficiency, a 100 Mbps link cannot transfer 50 TB in two weeks due to the fundamental bandwidth limitation.

How to eliminate wrong answers

Option A is wrong because AWS Storage Gateway (Volume Gateway) provides hybrid cloud storage with low-latency access via iSCSI or NFS, but it still relies on the internet connection for data transfer, which cannot meet the two-week deadline for 50 TB over 100 Mbps. Option C is wrong because Amazon S3 Transfer Acceleration uses AWS edge locations to optimize TCP transfers over the internet, but it does not increase the available bandwidth; the theoretical minimum transfer time for 50 TB at 100 Mbps is ~48 days, so acceleration cannot reduce it to two weeks. Option D is wrong because AWS DataSync over the internet uses the public internet or AWS Direct Connect, but even with optimization (e.g., parallel streams), the 100 Mbps bottleneck makes it impossible to transfer 50 TB within two weeks.

Practice this question →
345
MCQmedium

A company is streaming data from IoT devices to Amazon Kinesis Data Firehose, which writes to an Amazon S3 bucket. The data is then processed by an AWS Glue ETL job and loaded into Amazon Redshift. The team notices that some records are missing in Redshift. They suspect data loss during the Firehose delivery. Which configuration parameter should be checked first?

A.The AWS KMS key used for encryption.
B.The CloudWatch error logging configuration.
C.The buffer interval (e.g., 60 seconds) and buffer size.
D.The compression format (GZIP, Snappy, etc.).
AnswerC

Correct: If the buffer interval is too long and the stream is stopped, buffered data may be lost if not flushed properly.

Why this answer

Firehose can buffer data before writing to S3. If the buffer interval is too long and the stream ends, data may be lost if the buffer is not flushed. Option C (buffer interval) is the most likely cause.

Option A (compression) does not cause loss. Option B (KMS key) is for encryption. Option D (error logging) only logs errors, does not prevent loss.

Practice this question →
346
Multi-Selectmedium

Which TWO steps are required to set up cross-account access to an Amazon S3 data lake for AWS Glue jobs running in a different AWS account? (Choose two.)

Select 2 answers
A.Add a bucket policy to the S3 bucket that grants access to the Glue service role from the other account.
B.Create an IAM role in the second account that the Glue job can assume, with permissions to read from the S3 bucket.
C.Create a cross-account Glue crawler in the source account.
D.Set up VPC peering between the two accounts' VPCs.
E.Ensure both accounts are in the same AWS organization.
AnswersA, B

Correct: Bucket policy allows cross-account access.

Why this answer

An S3 bucket policy can grant cross-account access by specifying the AWS account ID of the second account as the principal, allowing the Glue service role from that account to read objects. This is a standard method for delegating access to S3 resources across accounts without requiring IAM roles in the source account.

Exam trap

The trap here is that candidates often confuse network-level connectivity (VPC peering) with IAM-level authorization, or assume that cross-account Glue crawlers are a built-in feature, when in fact the crawler must be in the same account as the data lake or use an assumed role with cross-account permissions.

Practice this question →
347
Multi-Selectmedium

A data engineer needs to design a data ingestion pipeline that ingests data from a MySQL database hosted on-premises into Amazon S3 for analytics. The pipeline must capture change data (CDC) and run continuously with low latency. Which two services should the data engineer use?

Select 2 answers
A.AWS Database Migration Service (DMS) with ongoing replication.
B.Amazon S3 as the target endpoint for DMS.
C.Amazon AppFlow.
D.AWS Glue ETL jobs scheduled at regular intervals.
E.Amazon Kinesis Data Streams.
AnswersA, B

DMS supports CDC and can write changes to S3 continuously.

Why this answer

AWS Database Migration Service (DMS) with ongoing replication can continuously capture changes from on-premises MySQL using Change Data Capture (CDC). Amazon S3 can be configured as the target endpoint for DMS, allowing the CDC data to be written directly to S3 with low latency. Option C (Amazon AppFlow) is designed for SaaS applications, not on-premises databases.

Option D (AWS Glue ETL) is batch-oriented and not suitable for low-latency continuous ingestion. Option E (Amazon Kinesis Data Streams) is not required because DMS can directly write to S3.

Practice this question →
348
Multi-Selectmedium

A data engineering team is designing a data lake on AWS. They need to store raw data in S3 and allow multiple analytics services to query the data. Which service can be used to catalog and provide schema information for the data?

Select 1 answer
A.AWS Glue Data Catalog
B.Amazon Kinesis Data Streams
C.Amazon RDS
D.Amazon DynamoDB
E.Amazon Athena
AnswersA

Glue Data Catalog stores metadata and schemas.

Why this answer

AWS Glue Data Catalog is a fully managed metadata repository that stores table definitions, schema information, and partition details for data in S3. Amazon Athena, while it can query data in S3 using SQL, does not provide its own catalog; it relies on the Glue Data Catalog for schema information. Therefore, only AWS Glue Data Catalog directly catalogs and provides schema information.

Exam trap

The trap is that Amazon Athena can create and query tables using DDL statements, leading candidates to think it serves as a catalog. However, Athena stores its table definitions in the Glue Data Catalog, making the Data Catalog the actual schema repository. Thus, only AWS Glue Data Catalog is the correct service for cataloging and providing schema information.

Practice this question →
349
MCQmedium

A company is using AWS Glue to catalog metadata from various data sources. The crawler is configured to run daily. However, the catalog is not reflecting new partitions added to an S3 bucket during the day. What is the MOST likely cause?

A.The S3 bucket has insufficient permissions for the Glue crawler
B.The table schema has changed and the crawler does not update it
C.The crawler is not scheduled frequently enough to capture changes
D.The data format is not supported by AWS Glue
AnswerC

The crawler runs once a day, so it misses partitions added between runs.

Why this answer

The crawler is configured to run daily, but new partitions are being added to the S3 bucket throughout the day. Since the crawler only runs once per day, it will not detect and catalog those new partitions until its next scheduled run. To capture changes more frequently, the crawler schedule should be increased or an event-driven trigger (e.g., using Amazon S3 Events and AWS Lambda) should be implemented.

Exam trap

The trap here is that candidates may assume the crawler automatically detects all changes in real time, but AWS Glue crawlers are batch-oriented and only discover new partitions during a crawl run, so scheduling frequency is critical.

How to eliminate wrong answers

Option A is wrong because if the S3 bucket had insufficient permissions for the Glue crawler, the crawler would fail entirely or produce errors, not selectively miss new partitions while still cataloging existing data. Option B is wrong because the question states that new partitions are not being reflected, not that the table schema has changed; Glue crawlers can update schemas by default unless configured otherwise, and schema changes would cause different symptoms (e.g., type mismatches). Option D is wrong because AWS Glue supports a wide range of data formats (CSV, JSON, Parquet, Avro, ORC, etc.), and if the format were unsupported, the crawler would fail to read the data entirely, not just miss new partitions.

Practice this question →
350
MCQeasy

A data engineer runs the AWS CLI command above to inspect a file in S3. They need to determine if the file was modified after a Glue ETL job processed it. What additional information could they obtain from this command?

A.The object's content type.
B.The object's storage class.
C.The object's last modified timestamp.
D.The object's ETag.
AnswerC

The LastModified field indicates when the object was last modified.

Why this answer

The AWS CLI command `aws s3api head-object --bucket my-bucket --key my-key` returns metadata about an S3 object without downloading it. The `LastModified` field in the response provides the exact timestamp of the last modification, which can be compared to the Glue ETL job's execution time to determine if the file was modified after processing. This is the only field that directly answers the question about modification timing.

Exam trap

The trap here is that candidates may confuse the `ETag` (a content hash) with a modification timestamp, or assume that `head-object` returns only basic metadata like size and type, overlooking the `LastModified` field that directly answers the question.

How to eliminate wrong answers

Option A is wrong because the `ContentType` field indicates the MIME type (e.g., text/csv) but has no relation to modification timing. Option B is wrong because the `StorageClass` field (e.g., STANDARD, GLACIER) describes the storage tier, not when the object was last changed. Option D is wrong because the `ETag` field is an MD5 hash (or a hash of concatenated parts for multipart uploads) used for integrity checks, not for tracking modification timestamps.

Practice this question →
← PreviousPage 5 of 5 · 350 questions total

Ready to test yourself?

Try a timed practice session using only Ml Data Engineering questions.

Start 20-question session