MLS-C01 Data Engineering Practice Test 24 — 15 Questions

Question 1

A company uses Amazon Kinesis Data Streams to ingest clickstream data. The stream has 8 shards. A Lambda function processes each record and writes to Amazon DynamoDB. The Lambda function sometimes fails due to DynamoDB write throttling, causing duplicate processing of records after retries. The data engineering team needs to ensure exactly-once processing semantics for the DynamoDB writes. What should the team do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use DynamoDB conditional writes with the Kinesis sequence number as a unique attribute to make writes idempotent.. Using DynamoDB conditional writes with the Kinesis sequence number as a unique attribute ensures idempotency. When processing a record, the Lambda function can attempt a conditional write that only succeeds if an item with that sequence number does not already exist. If the write fails due to a condition check, it means the record was already processed, so the function can skip it. This achieves exactly-once semantics even if the same record is delivered multiple times due to retries. Option A is incorrect because an SQS FIFO queue between Kinesis and Lambda would add latency and complexity; Kinesis already provides ordering and at-least-once delivery, and using a FIFO queue does not guarantee that the Lambda function will not process the same record multiple times within its retry logic. Option B is incorrect because setting maximum retry count to 0 with a DLQ simply discards failed records after the first failure, which does not provide exactly-once processing; it results in at-most-once semantics and potential data loss. Option C is incorrect because increasing DynamoDB write capacity may reduce throttling but does not eliminate duplicate processing; the same record can still be retried and written multiple times if the function retries, leading to duplicate writes.

Answer

Use an Amazon SQS FIFO queue between Kinesis and Lambda to deduplicate records.

Answer

Configure the Lambda event source mapping with a maximum retry count of 0 and a DLQ.

Answer

Increase the DynamoDB write capacity units to avoid throttling.

Question 2

A data pipeline uses AWS Glue to crawl an S3 bucket and create a table in the AWS Glue Data Catalog. The data is in Parquet format with partitions by date. After a new partition is added to S3, the crawler runs but the new partition is not reflected in the table. What is the most likely cause?

Accepted Answer

The new partition folder does not follow the Hive-style partition naming convention expected by the crawler.. The most likely cause is that the new partition folder does not follow the Hive-style partition naming convention expected by the crawler. AWS Glue crawlers expect partition directories to be named in the format key=value (e.g., date=2023-01-01). If the partitions are named differently, the crawler will not recognize them as partitions. Option A is incorrect because Glue crawlers do not require Lambda triggers. Option B is incorrect because schema mismatch would cause a different error, not just missing partitions. Option D is incorrect because while there is a limit on partitions, it is high enough that 'too many partitions' is less likely than a naming issue.

Answer

The crawler requires an AWS Lambda trigger to be configured for new partitions.

Answer

The Parquet schema in the new partition does not match the existing table schema.

Answer

The S3 bucket has too many partitions, exceeding the Glue crawler limit.

Question 3

A company uses Amazon Kinesis Data Firehose to deliver data to an Amazon S3 bucket. The data is organized by year/month/day/hour. The team needs to ensure that all data is encrypted at rest in S3 using an AWS KMS customer managed key (CMK). Which configuration should the team implement?

Accepted Answer

In the Firehose delivery stream configuration, enable S3 destination encryption and select the customer managed KMS key.. The correct approach. Kinesis Data Firehose can be configured to encrypt data at rest in S3 using AWS KMS. In the Firehose delivery stream configuration, under S3 destination settings, you can enable encryption and select a customer managed KMS key. This ensures all data written to S3 is encrypted with that key. Option A is incorrect because S3 default encryption applies to objects uploaded directly to S3, but Firehose writes objects using its own IAM role and can override default encryption; configuring encryption in Firehose is the recommended way. Option B is incorrect because using a Lambda function to encrypt after delivery adds unnecessary complexity and latency; Firehose can encrypt natively. Option D is incorrect because a bucket policy denying PutObject without the correct KMS key would work but is not the simplest or most straightforward configuration; Firehose can handle encryption directly.

Answer

Configure the S3 bucket's default encryption to use the customer managed KMS key.

Answer

Use an AWS Lambda function to encrypt the data after it is delivered to S3.

Answer

Add a bucket policy that denies PutObject unless the request includes the correct KMS key.

Question 4

A company uses Amazon S3 to store historical transaction data in CSV format. The data is partitioned by transaction_date. A data analyst runs Amazon Athena queries that frequently filter on customer_id and transaction_date. The queries are slow and expensive. The team needs to improve query performance and reduce cost. Which combination of actions should the team take? (Choose TWO.)

Accepted Answer

Convert the data from CSV to Parquet format.. Converting CSV to Parquet (option C) improves performance because Parquet is a columnar storage format that reduces the amount of data scanned by Athena, especially when queries only select a subset of columns. It also uses efficient compression, reducing storage and data scanned. Reorganizing the partition order (option D) to have customer_id first (the most frequently filtered column) improves partition pruning, reducing the amount of data read. Option A (S3 Select pushdown) is not fully supported by Athena; Athena already uses S3 Select for certain formats, but it doesn't guarantee significant improvement and may not be applicable. Option B (JSON) is worse than CSV because JSON is typically larger and not columnar. Option E (increasing workers) is not applicable as Athena is serverless and automatically scales.

Answer

Enable S3 Select pushdown in Athena to reduce data transfer.

Answer

Convert the data to JSON format for better query performance.

Answer

Increase the number of Athena query workers.

Question 5

A company uses Amazon Kinesis Data Streams to ingest clickstream data. The stream has 8 shards. A Lambda function processes each record and writes to Amazon DynamoDB. The Lambda function sometimes fails due to DynamoDB write throttling, causing duplicate processing of records after retries. The data engineering team needs to ensure exactly-once processing semantics for the DynamoDB writes. What should the team do?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of shards in the Kinesis stream.. Increase the number of shards in the Kinesis stream. Kinesis Data Streams throughput is determined by the number of shards; each shard provides 1 MB/s or 1000 records/s for reading. Since the consumer is falling behind, increasing shards directly increases the read throughput. Option A is incorrect because adding more consumer instances without increasing shards will not improve throughput; KCL ensures each shard is processed by one worker, so extra workers are idle. Option B is incorrect because increasing DynamoDB write capacity may reduce throttling but does not address the root cause of low read throughput from Kinesis. Option C is incorrect because increasing the data retention period does not affect the rate at which data can be read.

Answer

Deploy additional consumer instances using the same application name.

Answer

Increase the write capacity of the DynamoDB table.

Answer

Increase the data retention period of the stream to 7 days.

Question 6

A company uses Amazon Redshift to run analytics on sales data. The data is loaded daily from S3 using COPY commands. The team notices that the COPY command performance degrades over time due to table bloat. The team needs to maintain query performance and reduce storage costs. Which combination of maintenance operations should the team perform regularly? (Choose THREE.)

Accepted Answer

Run the VACUUM command to reclaim space and re-sort rows.. The correct maintenance operations are VACUUM, DEEP COPY, and ANALYZE. VACUUM reclaims space from deleted or updated rows and re-sorts data if sort keys are defined, reducing bloat. DEEP COPY recreates the table to eliminate bloat completely by copying data to a new table and renaming. ANALYZE updates table statistics, which helps the query planner optimize query performance. Option A (UNLOAD) is wrong because it exports data to S3, not a maintenance operation. Option B (changing distribution style) is a schema change that affects data distribution, not a regular maintenance task for bloat removal.

Answer

Run the UNLOAD command to export data to S3 and then reload.

Answer

Change the distribution style of the table to KEY.

Question 7

A data engineer needs to transfer 50 TB of data from an on-premises HDFS cluster to Amazon S3. The data must be encrypted in transit and at rest. The on-premises network has a 1 Gbps connection to AWS. The transfer must complete within 5 days. Which solution is MOST cost-effective and meets the requirements?

Accepted Answer

Use AWS DataSync with a DataSync agent installed on-premises to transfer the data to S3.. (AWS DataSync). With a 1 Gbps connection, the maximum theoretical transfer in 5 days is about 54 TB (1 Gbps = 0.125 GB/s, 0.125 * 86400 * 5 = 54000 GB = 54 TB), so network transfer is feasible within the time limit. AWS DataSync uses a DataSync agent installed on-premises to transfer data from HDFS to S3, encrypting data in transit (TLS) and at rest (S3 server-side encryption). This is the most cost-effective solution because it avoids the hardware and shipping costs of Snowball Edge (option C). Option A (S3 Transfer Acceleration) does not directly integrate with HDFS and is designed for speeding up uploads over public internet, not for encrypting data from HDFS. Option D (AWS Glue) is an ETL service, not a data transfer solution, and would require additional infrastructure and complexity.

Answer

Use S3 Transfer Acceleration to upload the data directly from HDFS to S3.

Answer

Order an AWS Snowball Edge device and copy the data to it, then ship it back.

Answer

Use AWS Glue to read from HDFS and write to S3 in a continuous ETL job.

Question 8

A company uses AWS Glue to run ETL jobs on a daily basis. The jobs read from Amazon RDS and write to Amazon S3. The data volume has grown, and the jobs are taking longer to complete. The team wants to optimize the jobs for cost and performance. Which combination of techniques should the team implement? (Choose THREE.)

Accepted Answer

Use a larger Glue worker type, such as G.2X, for more memory per worker.. (larger worker type) provides more memory and CPU per worker, improving performance for heavy workloads. Option B (job bookmarks) enables incremental processing, reducing the amount of data read on subsequent runs. Option E (pushdown predicates) filters data at the source in the JDBC connection, reducing data transferred across the network. Option C is incorrect because increasing partitions in the output S3 data does not affect the processing speed of the current job. Option D is incorrect because increasing the maximum number of DPUs increases cost linearly and may not be as effective as using larger workers or other optimizations.

Answer

Increase the number of partitions in the output S3 data to improve parallelism.

Answer

Increase the maximum number of DPUs for the job to 100.

Question 9

A company uses Amazon EMR to run Spark jobs on a cluster with 10 core nodes of type r5.xlarge. The jobs are I/O intensive and read large amounts of data from S3. The team notices high network throughput but low CPU utilization. Which configuration change would improve job performance at the same cost?

Accepted Answer

Replace the core nodes with r5d.xlarge instances that have local SSDs.. R5d instances include local NVMe SSDs. These SSDs can be used for caching intermediate data during Spark jobs, reducing the need to read from and write to S3 over the network. This directly addresses the I/O bottleneck and high network throughput observed, improving job performance. Option A is incorrect because moving to general-purpose m5 instances does not provide local SSDs and may not improve I/O. Option B is incorrect because doubling the number of core nodes would increase cost significantly without necessarily solving the I/O issue. Option D is incorrect because spot instances reduce cost but do not inherently improve I/O performance; they may even add instability.

Answer

Change the instance type to m5.xlarge (general purpose) to balance resources.

Answer

Increase the number of core nodes to 20.

Answer

Use spot instances for the core nodes to save cost and reinvest in more nodes.

Question 10

Refer to the exhibit. A data engineer has attached this IAM policy to an IAM role used by an AWS Glue ETL job. The job reads from an S3 bucket (data-bucket) that is encrypted with SSE-KMS using the key arn:aws:kms:us-east-1:123456789012:key/abc123, transforms the data, and writes the result to a different S3 bucket (output-bucket) encrypted with a different KMS key (arn:aws:kms:us-east-1:123456789012:key/xyz789). When the job runs, it fails with an access denied error. What is the cause?

Accepted Answer

The policy does not grant kms:Encrypt permission for the output bucket's KMS key.. The job fails because the IAM policy grants kms:Decrypt and kms:GenerateDataKey for the input bucket's KMS key (abc123) but does not grant kms:Encrypt or kms:GenerateDataKey for the output bucket's KMS key (xyz789). To write encrypted data to the output bucket, the AWS Glue job must have permission to encrypt using the output KMS key. Option D is correct because the missing kms:Encrypt permission causes the access denied error. Option A is incorrect because the policy includes s3:GetObject for the input bucket. Option B is incorrect because Glue catalog permissions are not relevant to the encryption error. Option C is incorrect because the error is due to missing KMS permissions for the output bucket, not because the policy does not include s3:PutObject.

Answer

The policy does not include s3:GetObject permission for the output bucket.

Answer

The policy does not include glue:CreateTable permission.

Answer

The policy does not include s3:PutObject permission for the output bucket.

Question 11

A company uses Amazon Kinesis Data Firehose to deliver data to Amazon S3. The data must be transformed before delivery using AWS Lambda. The Lambda function adds a timestamp field. The Firehose stream receives up to 10,000 records per second. The transformation currently takes 500 ms per record. What should the team do to ensure the transformation can keep up with the incoming data without data loss?

Accepted Answer

Increase the Lambda concurrency limit for the function to handle parallel invocations.. Increasing the Lambda concurrency limit allows more parallel invocations, enabling the function to process the high throughput of 10,000 records per second (each taking 500 ms) without falling behind. Option A is incorrect because shards are a concept for Kinesis Data Streams, not Firehose. Option B is incorrect: placing the Lambda function in a VPC typically adds network latency and does not improve performance for this simple transformation. Option D is incorrect: increasing S3 buffer size/interval may delay data delivery but does not increase transformation capacity.

Answer

Increase the number of shards in the Kinesis stream.

Answer

Place the Lambda function in a VPC to improve network performance.

Answer

Increase the S3 buffer size and buffer interval in the Firehose delivery stream.

Question 12

A company runs a critical data pipeline using Apache Spark on Amazon EMR. The pipeline reads data from Amazon S3, performs complex transformations, and writes results back to S3. The job runs every hour and must complete within 30 minutes. Recently, the job has been taking longer and occasionally failing due to executor losses. The team suspects memory pressure. Which action should the team take to improve stability and performance without increasing cost?

Accepted Answer

Enable Spark dynamic allocation to adjust executors based on workload.. Enabling Spark dynamic allocation allows the cluster to automatically scale the number of executors up and down based on the workload. This helps alleviate memory pressure by releasing idle executors and requesting additional executors only when needed, improving resource utilization without increasing overall cluster cost. Option A is incorrect because simply increasing spark.executor.memory may cause YARN container failures if the instance memory is exceeded, and does not address the root cause of executor losses. Option B is incorrect because adding core nodes increases cost and may not resolve memory pressure if the issue is inefficient resource allocation. Option C is incorrect because decreasing shuffle partitions reduces parallelism and can increase memory per task, potentially worsening memory pressure and prolonging job runtime.

Answer

Increase the spark.executor.memory setting to allocate more memory per executor.

Answer

Increase the number of core nodes in the EMR cluster.

Answer

Decrease the number of shuffle partitions (spark.sql.shuffle.partitions) to reduce overhead.

Question 13

A company runs a real-time analytics platform that ingests IoT sensor data from millions of devices. The data is sent to Amazon Kinesis Data Streams with 16 shards. A custom Java application using the Kinesis Client Library (KCL) processes the data and writes aggregated results to Amazon DynamoDB. The application runs on a fleet of EC2 instances in an Auto Scaling group. Recently, the team noticed that some records are being processed multiple times, resulting in duplicate entries in DynamoDB. The application uses the DynamoDB PutItem API to write records. The team needs to eliminate duplicates without significantly increasing latency. Which solution should the team implement?

Accepted Answer

Use DynamoDB TransactWriteItems with a condition check that the record's Kinesis sequence number does not already exist in the table.. Using a DynamoDB transaction with a condition check on the Kinesis sequence number ensures that each record is written only once. Option A is wrong because increasing write capacity does not address duplicate processing; duplicates arise from the KCL consumer processing records multiple times, not from throttling. Option C is wrong because while SQS FIFO provides deduplication at the queue level, it does not guarantee exactly-once processing downstream in DynamoDB; the consumer could still write duplicates if it fails after writing but before deleting the message. Additionally, adding an extra queue increases latency and complexity. Option D is wrong because BatchWriteItem does not decrease duplicates; it only batches multiple put requests into one API call and still requires idempotency measures.

Answer

Enable DynamoDB auto scaling to increase write capacity and reduce throttling, which causes retries and duplicates.

Answer

Place an Amazon SQS FIFO queue between the KCL application and DynamoDB to deduplicate messages.

Answer

Modify the application to use DynamoDB BatchWriteItem instead of PutItem to reduce the number of write requests.

Question 14

A company uses AWS Glue to run ETL jobs that process data from Amazon RDS for PostgreSQL and load it into Amazon Redshift. The Glue job runs nightly and takes 6 hours to complete. The Redshift cluster is a single dc2.large node. The team needs to reduce the load time to under 3 hours. The data volume is 200 GB per night. The team is considering using Amazon Redshift Spectrum to query data directly from S3 instead of loading it. However, the data transformation logic is complex and requires multiple joins and aggregations that are currently performed in Glue. Which approach should the team recommend to meet the time requirement?

Accepted Answer

Increase the Redshift cluster to a multi-node cluster with dc2.8xlarge nodes to improve COPY and query performance.. Increasing the Redshift cluster to a multi-node configuration with dc2.8xlarge nodes significantly improves COPY performance by parallelizing data loading across multiple slices. The current single dc2.large node may be the bottleneck for the data loading portion of the Glue job. Faster loading can reduce the total 6-hour runtime to under 3 hours if loading is the dominant factor. Note: the complex transformations remain in AWS Glue, not on Redshift. Option A is incorrect because Redshift Spectrum allows querying external tables without loading but does not accelerate the complex ETL transformations. Option C splits the load but does not address the single-node ingestion bottleneck and adds complexity. Option D improves load speed with Parquet but the cluster's ingestion capacity is still the limiting factor.

Answer

Use Redshift Spectrum to create external tables and run the transformations directly in Redshift, bypassing the Glue job.

Answer

Split the Glue job into multiple parallel jobs that each load a portion of the data into separate Redshift tables, then use UNION ALL views.

Answer

Stage the data in S3 in Parquet format and use a COPY command with the PARQUET option to load data faster.

Question 15

A company uses Amazon S3 to store log files from various applications. The logs are in JSON format and are appended to existing files every few minutes. A data analyst wants to run SQL queries on the logs using Amazon Athena. However, queries return incomplete results because Athena does not support modifying data. The team needs to enable querying of the latest log data with minimal changes to the existing ingestion process. Which solution should the team implement?

Accepted Answer

Create an Athena table using the Hive JSON SerDe that reads the logs directly from the existing S3 bucket.. Athena supports reading JSON data with the Hive JSON SerDe. By creating a table with the appropriate SerDe, the analyst can query the JSON logs directly without modifying the ingestion process. Option A is incorrect because converting to Parquet would require changing the ingestion process. Option B is incorrect because using Kinesis Data Firehose would require altering the ingestion pipeline. Option D is incorrect because loading into Redshift adds complexity and latency.

Answer

Convert the logs to Parquet format using a scheduled AWS Glue job and store them in a separate S3 bucket.

Answer

Stream the logs to Amazon Kinesis Data Firehose, which writes the data to S3 in Parquet format.

Answer

Use AWS Glue to load the JSON logs into Amazon Redshift and query using Redshift.