MLS-C01 Data Engineering Practice Test 17 — 15 Questions

Question 1

A data engineer runs the AWS CLI command above to inspect an object in S3. The engineer wants to query this metadata (kafka-offset) using Amazon Athena to track processing progress. How can the engineer make this metadata available for Athena queries without modifying the existing data pipeline?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use AWS Glue to create a table that includes the metadata as a column by running an ETL job.. AWS Glue ETL jobs can read the S3 object's user-defined metadata (e.g., 'kafka-offset') and write it as a column in a new or transformed dataset, which Athena can then query. This approach does not modify the existing data pipeline, as the original objects remain unchanged; the metadata is extracted and stored in a queryable format (e.g., Parquet or CSV) in a separate location. Glue's ability to access S3 object metadata via the `getObjectMetadata` API during ETL processing makes this a clean, pipeline-agnostic solution.

Answer

Use S3 object tags instead of metadata and query the tags using Athena.

Answer

Use an AWS Lambda function to copy the metadata into the object's content as a new line.

Answer

Use Amazon Athena to query the object metadata directly by referencing the metadata field.

Question 2

A data engineer configures an S3 event notification to trigger an AWS Lambda function when a new object is created in 'my-input-bucket'. The Lambda function processes the CSV file and writes results to 'my-output-bucket'. The engineer notices that the Lambda function is not triggered for some objects. Which step should the engineer take to diagnose the issue?

Accepted Answer

Check the Lambda function's resource-based policy to ensure S3 has permission to invoke the function.. The most likely cause of the Lambda function not being triggered for some objects is that S3 lacks the necessary permission to invoke the function. S3 event notifications require a resource-based policy (also known as a Lambda function policy) that explicitly grants the S3 service principal permission to invoke the function. Without this policy, S3 will not be able to trigger the Lambda function, even if the event notification configuration is correct.

Answer

Check the Lambda function's execution role for permissions to write to the output bucket.

Answer

Review the CloudWatch Logs for the Lambda function to see if there are errors.

Answer

Verify that the S3 event notification is configured with the correct prefix and suffix filters.

Question 3

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Kinesis Data Analytics application that runs SQL queries. The application has been failing intermittently with 'ProvisionedThroughputExceededException' errors. Which action should be taken to resolve this issue?

Accepted Answer

Increase the number of shards in the Kinesis data stream.. The 'ProvisionedThroughputExceededException' error indicates that the Kinesis Data Stream's read or write throughput limits have been exceeded. Increasing the number of shards in the stream directly increases the total provisioned throughput, allowing the Kinesis Data Analytics application to consume data without throttling.

Answer

Disable error logging in the Kinesis Data Analytics application.

Answer

Increase the record size in the Kinesis data stream.

Answer

Switch from Kinesis Data Analytics to Kinesis Data Firehose.

Question 4

A data engineering team is designing a data pipeline to process large CSV files (10-50 GB each) stored in Amazon S3. The pipeline must transform the data using AWS Glue and load it into Amazon Redshift for analytics. The team wants to minimize costs while ensuring the pipeline can handle peak loads. Which approach is the most cost-effective?

Accepted Answer

Use AWS Glue with Spark and dynamic frames, scaling the number of workers based on file size.. AWS Glue with Spark and dynamic frames is the most cost-effective approach because it is serverless, automatically scales workers based on file size, and is optimized for ETL on large CSV files (10-50 GB) in S3. Dynamic frames provide built-in transformations and schema inference, reducing development effort, while the ability to adjust the number of workers allows handling peak loads without over-provisioning. This minimizes idle compute costs compared to always-on clusters like EMR.

Answer

Use AWS Lambda to process each file and load into Redshift.

Answer

Use Amazon EMR with Hive to transform the data and load into Redshift.

Answer

Use an AWS Glue Python shell job with a single r5.xlarge worker.

Question 5

A company is using Amazon DynamoDB to store sensor data. The data is exported to Amazon S3 using DynamoDB Streams and AWS Lambda for long-term archival. Recently, the Lambda function has been failing due to 'ProvisionedThroughputExceededException' on the DynamoDB stream. What is the most likely cause?

Accepted Answer

The Lambda function is processing records too slowly, causing the stream to throttle.. A is correct because the 'ProvisionedThroughputExceededException' on a DynamoDB stream indicates that the stream's read throughput is being throttled. When a Lambda function processes records too slowly, it cannot keep up with the rate of new records being written to the stream, causing the stream shards to throttle the Lambda consumer. This is a common issue when the Lambda function's processing time per record is high or when the function is invoked with a large batch size that exceeds its processing capacity.

Answer

The DynamoDB stream is disabled.

Answer

The DynamoDB table's write capacity is too low.

Answer

The Lambda function does not have enough memory allocated.

Question 6

A data scientist is building a training dataset from data stored in Amazon S3. The data consists of JSON files each containing a 'timestamp' field. The scientist wants to use AWS Glue to catalog the data and enable querying via Amazon Athena. However, Athena queries are returning zero results for time-range filters. What is the most likely cause?

Accepted Answer

The JSON files are not in the correct format for Athena.. Athena supports querying JSON data, but the JSON files must have a schema that Athena can interpret. If the 'timestamp' field is in an unrecognized date/time format or the JSON structure is inconsistent, Athena may fail to parse the data correctly, resulting in zero rows for time-range filters. Option C is correct because the most likely cause is the JSON files not being in a format that Athena can parse properly for timestamp filtering. Option A is wrong because permission issues would cause an access denied error. Option B is wrong because Athena supports nested JSON. Option D is wrong because even if the timestamp is not a partition column, filtering on it should still return results if the data matches the filter condition.

Answer

The AWS Glue crawler does not have permissions to read the S3 bucket.

Answer

Athena cannot query nested JSON objects.

Answer

The 'timestamp' field is not defined as a partition column in the Glue table.

Question 7

A company is streaming data from IoT devices to Amazon Kinesis Data Firehose, which writes to an Amazon S3 bucket. The data is then processed by an AWS Glue ETL job and loaded into Amazon Redshift. The team notices that some records are missing in Redshift. They suspect data loss during the Firehose delivery. Which configuration parameter should be checked first?

Accepted Answer

The buffer interval (e.g., 60 seconds) and buffer size.. Firehose can buffer data before writing to S3. If the buffer interval is too long and the stream ends, data may be lost if the buffer is not flushed. Option C (buffer interval) is the most likely cause. Option A (compression) does not cause loss. Option B (KMS key) is for encryption. Option D (error logging) only logs errors, does not prevent loss.

Answer

The AWS KMS key used for encryption.

Answer

The CloudWatch error logging configuration.

Answer

The compression format (GZIP, Snappy, etc.).

Question 8

A data engineer needs to set up a data pipeline that ingests data from an Amazon RDS MySQL database into Amazon S3. The pipeline should run daily and capture incremental changes (inserts, updates, deletes) from the source database. Which AWS service should be used as the data ingestion tool?

Accepted Answer

AWS Database Migration Service (DMS) with continuous change data capture (CDC).. AWS DMS with continuous CDC is the correct choice because it is specifically designed to capture incremental changes (inserts, updates, deletes) from a relational database like Amazon RDS MySQL and replicate them to Amazon S3. DMS uses the MySQL binary log (binlog) to track row-level changes in near real-time, making it ideal for daily incremental pipelines. Other services either lack native CDC support or are not optimized for database-to-object-store incremental ingestion.

Answer

Amazon Kinesis Data Streams with a Lambda function.

Answer

AWS Data Pipeline with a SQL activity.

Answer

AWS Glue with a scheduled crawler.

Question 9

A company is building a data lake on Amazon S3 and wants to use AWS Glue to catalog the data. The data includes CSV, Parquet, and JSON files. The team wants to ensure that the Glue crawler can infer the schema correctly and update the Data Catalog when new partitions are added. Which crawler configuration should be used?

Accepted Answer

Create a single crawler that includes all file extensions and set the 'Update all new and existing partitions' option.. A single AWS Glue crawler can handle multiple file formats (CSV, Parquet, JSON) in a data lake on Amazon S3, and enabling 'Update all new and existing partitions' ensures the Data Catalog is refreshed with both new partitions and any schema changes in existing partitions. This configuration maintains a consistent and up-to-date catalog without manual intervention, which is essential for downstream analytics and machine learning workloads.

Answer

Create separate crawlers for each file format and schedule them at different times.

Answer

Use a crawler that only catalogs Parquet files because they are more efficient.

Answer

Use a crawler with 'Update all new and existing partitions' disabled to avoid schema conflicts.

Question 10

A company uses Amazon Kinesis Data Streams with a shard count of 5. The data producer sends 1000 records per second, each 1 KB in size. The consumer application reads from the stream using the Kinesis Client Library (KCL) and processes records. The consumer is experiencing high latency and falling behind. What is the most effective way to improve consumer throughput?

Accepted Answer

Increase the number of shards in the stream.. The consumer is falling behind because it cannot process records fast enough. The Kinesis Client Library (KCL) typically runs one consumer thread per shard, so with only 5 shards there are only 5 parallel consumers. To improve throughput, the number of shards should be increased to allow more parallel processing. Options like enhanced fan-out would improve read throughput per consumer but do not increase parallelism; the bottleneck here is processing speed, not read throughput. Increasing shards directly increases the number of consumers and thus overall throughput.

Answer

Switch to Kinesis Data Analytics for processing.

Answer

Use enhanced fan-out to dedicate read throughput to the consumer.

Answer

Increase the record size to 5 KB.

Question 11

A company wants to store semi-structured data from IoT sensors in a cost-effective manner for occasional querying. The data is not updated once written. Which Amazon S3 storage class is the most cost-effective for this use case?

Accepted Answer

S3 Glacier Deep Archive. S3 Glacier Deep Archive is the most cost-effective storage class for semi-structured IoT sensor data that is written once and only occasionally queried. It offers the lowest storage cost among S3 classes (approximately $0.00099/GB/month), making it ideal for long-term archival of immutable data where retrieval times of 12–48 hours are acceptable.

Answer

S3 Standard

Answer

S3 One Zone-Infrequent Access

Answer

S3 Intelligent-Tiering

Question 12

Which TWO configurations are required to enable AWS Glue to access data stored in a VPC? (Choose two.)

Accepted Answer

A VPC endpoint for Amazon S3.. A VPC endpoint for Amazon S3 (Option A) is required because it allows AWS Glue to access S3 data privately over the AWS network without traversing the public internet, which is necessary when Glue runs inside a VPC. An AWS Glue connection object (Option B) is required because it defines the VPC, subnet, and security group that Glue will use to launch its resources within the VPC, enabling it to access data stores in that VPC.

Answer

A NAT gateway in a public subnet.

Answer

An Internet gateway attached to the VPC.

Answer

An S3 bucket policy that allows access from the Glue service principal.

Question 13

Which THREE factors should be considered when choosing between Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose for a real-time data ingestion pipeline? (Choose three.)

Accepted Answer

The need for custom real-time processing logic using consumer applications.. Kinesis Data Streams supports custom real-time processing via consumer applications using the Kinesis Client Library (KCL) or AWS Lambda, enabling fine-grained control over record processing, checkpointing, and custom logic. This is a key differentiator from Kinesis Data Firehose, which only supports built-in transformations via Lambda and does not allow direct consumer access to the stream.

Answer

The need for built-in data transformation and analytics.

Answer

The requirement for exactly-once delivery semantics.

Question 14

Which TWO steps are required to set up cross-account access to an Amazon S3 data lake for AWS Glue jobs running in a different AWS account? (Choose two.)

Accepted Answer

Add a bucket policy to the S3 bucket that grants access to the Glue service role from the other account.. An S3 bucket policy can grant cross-account access by specifying the AWS account ID of the second account as the principal, allowing the Glue service role from that account to read objects. This is a standard method for delegating access to S3 resources across accounts without requiring IAM roles in the source account.

Answer

Create a cross-account Glue crawler in the source account.

Answer

Set up VPC peering between the two accounts' VPCs.

Answer

Ensure both accounts are in the same AWS organization.

Question 15

Refer to the exhibit. A company is using the Kinesis stream 'my-stream' with one shard. The producer is sending 1000 records per second, each 1 KB. The consumer is reading from the stream using the Kinesis Client Library (KCL). The consumer is able to process 500 records per second per shard. What is the most likely cause of the consumer falling behind?

Accepted Answer

The stream uses KMS encryption, which adds latency.. The consumer is processing 500 records per second per shard, which is half the producer's rate of 1000 records per second. While the shard's read throughput limit is 2 MB/s, not 1 MB/s, the consumer's processing speed is the bottleneck. Among the options, KMS encryption is the most plausible cause because decryption latency can significantly slow down the consumer, especially if the consumer application is not optimized for encryption overhead.

Answer

The retention period is set to 24 hours, which is too short.

Answer

The stream has only one shard, which limits the read throughput to 1 MB/s.

Answer

The consumer application is not using enhanced fan-out.