MLS-C01 Data Engineering Practice Test 23 — 15 Questions

Question 1

A data scientist is using Amazon SageMaker to train a model. The training data is stored in Amazon S3 and is approximately 500 GB. The data scientist notices that the training job is taking a long time to start because the data is being copied to the training instance's storage. The data scientist wants to reduce the startup time for subsequent training jobs. Which action should the data scientist take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Pipe input mode instead of File input mode for the training job. Using Pipe input mode streams data directly from S3 to the training algorithm without downloading, reducing startup time. Option B is wrong because FSx for Lustre is not needed for simple streaming. Option C is wrong because increasing instance storage does not address the data transfer issue. Option D is wrong because using EBS optimized instances does not change the data loading mechanism.

Answer

Use an EBS-optimized instance type

Answer

Use Amazon FSx for Lustre as a high-performance file system mounted to the training instance

Answer

Increase the size of the training instance's Amazon EBS storage volume

Question 2

A company is designing a data pipeline to process log files from multiple sources. The logs are written to Amazon S3 every hour. The data is then transformed using AWS Glue ETL jobs and loaded into Amazon Redshift for analysis. The company needs to ensure that the data is available for analysis within 30 minutes of being written to S3. Currently, the Glue job is triggered hourly, but the company wants to reduce the latency. Which solution should the company implement?

Accepted Answer

Use Amazon S3 event notifications to invoke an AWS Lambda function that starts the Glue job automatically. Configuring an S3 event notification to invoke AWS Lambda, which starts the Glue job, allows near-real-time processing within minutes. Option A is wrong because hourly triggers do not reduce latency. Option B is wrong because increasing the crawler frequency does not trigger ETL jobs. Option D is wrong because Redshift Spectrum does not transform data.

Answer

Increase the frequency of the Glue crawler to run every 5 minutes

Answer

Use Amazon Redshift Spectrum to query the data directly from S3 without transformation

Answer

Reduce the Glue job trigger frequency to every 15 minutes

Question 3

A company is building a data pipeline that ingests data from on-premises databases into Amazon S3 using AWS Database Migration Service (AWS DMS). The company wants to capture continuous changes from the source database and replicate them to S3 in near-real time. Which AWS DMS configuration should the company use?

Accepted Answer

Create a CDC-only task with S3 as the target endpoint. Using a CDC-only task with S3 as the target endpoint replicates continuous changes to S3. Option A is wrong because a full-load task only migrates existing data. Option B is wrong because a full-load plus CDC task includes both, but the requirement is only changes. Option C is wrong because a validation task is for data validation, not replication.

Answer

Create a full-load task to copy the existing data

Answer

Create a full-load plus CDC task with S3 target

Answer

Create a validation task to compare source and target

Question 4

A data engineering team is designing a data lake on Amazon S3. The data is ingested from multiple sources in JSON, CSV, and Parquet formats. The team needs to make the data available for analysis using Amazon Athena and Amazon Redshift Spectrum. The team wants to minimize data transformation costs and storage overhead. Which data storage approach should the team use?

Accepted Answer

Convert all data to Apache Parquet before storing in S3. Converting all data to Apache Parquet before storing in S3 minimizes storage overhead and improves query performance. Parquet is a columnar format that provides efficient compression and encoding schemes, reducing storage costs. It is natively supported by Amazon Athena and Redshift Spectrum, enabling fast analytics without on-the-fly conversion. Option B (storing in original format) increases storage costs and can degrade query performance, especially with JSON or CSV. Option C incurs transformation costs each time data is queried, negating any storage benefit. Option A adds unnecessary transformation steps and cluster costs. Therefore, upfront conversion to Parquet is the most cost-effective strategy for this use case.

Answer

Load the data into Amazon Redshift cluster and then unload to S3 in Parquet

Answer

Store the data in its original format in S3 and use Athena to query directly

Answer

Store the data in its original format and use AWS Glue to convert to Parquet when queried

Question 5

A company uses Amazon Kinesis Data Streams to ingest clickstream data from a website. The data is consumed by a custom application that runs on Amazon EC2 instances. The company notices that the consumer application is falling behind the producer, causing data to be throttled. Which action should the company take to improve the consumer's throughput?

Accepted Answer

Increase the number of shards in the Kinesis data stream. Increasing the number of shards increases the stream's read capacity, allowing more consumers to read in parallel and improving throughput. Option A is wrong because reducing the data retention period does not increase read throughput; it only affects how long data is stored. Option C is wrong because Lambda concurrency is applicable only to Lambda functions, not to the custom EC2 application consuming the stream. Option D is wrong because Amazon Kinesis Data Firehose is a different service for delivering streaming data to destinations like S3, and it does not improve the throughput of the existing EC2 consumer.

Answer

Reduce the data retention period of the stream

Answer

Increase the maximum concurrency of the AWS Lambda function that processes the stream

Answer

Use Amazon Kinesis Data Firehose to deliver data to Amazon S3

Question 6

A company is using AWS Glue to run ETL jobs that process data from Amazon RDS to Amazon S3. The ETL jobs are failing intermittently with write timeout errors when writing to S3. The company wants to implement a retry mechanism for transient errors. What should the company do?

Accepted Answer

Configure the AWS Glue job to retry on failure by setting the 'Max retries' parameter. AWS Glue provides a built-in retry mechanism via the 'Max retries' parameter, which automatically retries the job when transient errors like write timeouts to S3 occur. Option A directly addresses the need for a retry mechanism. Option B is incorrect because increasing EBS volume size does not fix S3 write timeout errors, which are network-related. Option C is incorrect because manual restart via CloudWatch is not an automated retry mechanism. Option D is incorrect because AWS Glue does not natively integrate with SQS for job retries; the built-in retry parameter is the proper solution.

Answer

Increase the size of the Amazon EBS volumes attached to the Glue job

Answer

Use Amazon CloudWatch to monitor the job and manually restart on failure

Answer

Place the failed job messages in an Amazon SQS queue and reprocess them

Question 7

A company is migrating its on-premises Apache Hadoop cluster to AWS. The cluster processes large datasets using Spark jobs. The company wants to minimize operational overhead and use native AWS services. Which combination of services should the company use?

Accepted Answer

Amazon EMR with Spark and Amazon S3. Amazon EMR is a managed Hadoop framework that natively supports Spark jobs, and Amazon S3 provides scalable and durable object storage for the data. This combination minimizes operational overhead as EMR automatically handles cluster provisioning, scaling, and monitoring. Option B is incorrect because Amazon Redshift is a data warehouse, not a Hadoop cluster, and Spectrum is for querying data in S3, not for running Spark jobs. Option C is incorrect because Amazon Athena is a serverless query service for SQL-based analytics, not for executing Spark jobs, and AWS Glue is an ETL service, not a compute engine for Spark. Option D is incorrect because running Apache Spark on EC2 instances requires manual setup, maintenance, and scaling of the cluster, increasing operational overhead compared to using a managed service like EMR.

Answer

Amazon Redshift with Spectrum and Amazon S3

Answer

Amazon Athena and AWS Glue

Answer

Amazon EC2 instances with Apache Spark installed and Amazon S3

Question 8

A company is building a data pipeline that uses Amazon Kinesis Data Streams to ingest real-time events. The pipeline then uses AWS Lambda to process the events and store results in Amazon DynamoDB. The company wants to ensure that the Lambda function can process all events without data loss and without duplicating processing. Which TWO configuration steps should the company take?

Accepted Answer

Increase the data retention period of the Kinesis stream to 7 days to allow reprocessing. To ensure all events are processed without data loss and without duplicating processing, the company should increase the Kinesis stream retention to 7 days (option A). This allows reprocessing of events that fail initial processing, thus preventing data loss. The company should also configure the Lambda function to disable retries on failure (option E). Automatic retries can cause the same batch of events to be processed multiple times, leading to duplication unless the Lambda function is idempotent. By disabling retries, the company can handle failures manually, ensuring exactly-once processing when combined with idempotent reprocessing logic. Option B (reducing batch window) does not prevent duplicates and may increase invocations. Option C (enabling iterator age metric) is for monitoring, not preventing loss/duplication. Option D (single shard) does not prevent duplication and can cause throughput limitations.

Answer

Set the Lambda function's batch window to a small value (e.g., 1 second) to reduce processing latency

Answer

Enable the 'iterator age' metric in Amazon CloudWatch to monitor consumer lag

Answer

Use a single shard for the Kinesis stream to ensure order and avoid parallel processing issues

Question 9

A company is using AWS Glue to catalog data stored in Amazon S3. The data is partitioned by year, month, day, and hour. The company runs hourly ETL jobs that add new partitions. The Glue crawler is scheduled to run every hour to update the Data Catalog. However, the crawler is taking longer than expected and is not completing before the next crawler run starts. Which action could the company take to resolve this issue?

Accepted Answer

Use multiple crawlers, each configured to crawl a different path (e.g., one for year=2023, one for year=2024). Using multiple crawlers to crawl different paths allows parallel processing of partitions, reducing crawler time. Option A is incorrect because the 'Schema updates' option does not increase throughput. Option B is incorrect because partition indexing speeds up queries, not the crawler itself. Option C is incorrect because reducing frequency does not speed up the crawler. Option E is incorrect because AWS Glue does not have a 'Crawler queue' feature; to increase parallelism, you would increase DPUs.

Answer

Increase the throughput of the crawler by configuring the 'Schema updates' option

Answer

Enable partition indexing on the table to speed up the crawler

Answer

Decrease the crawler schedule frequency to every 2 hours to avoid overlapping runs

Answer

Increase the number of crawler instances by configuring the 'Crawler queue' to process multiple partitions in parallel

Question 10

A company needs to move 50 TB of data from an on-premises data center to Amazon S3. The company has a limited internet bandwidth of 100 Mbps. The data transfer must be completed within 10 days. Which TWO services should the company use together to meet these requirements?

Accepted Answer

Amazon S3 as the destination. The correct answers are A (Amazon S3 as the destination) and D (AWS Snowball Edge). The company needs to transfer 50 TB within 10 days, but with only 100 Mbps bandwidth, only about 10.8 TB can be transferred over the internet in that time. Therefore, a physical transfer solution like AWS Snowball Edge is required to move the data offline to S3. Option B (AWS Direct Connect) is incorrect because even though it provides a dedicated connection, it still requires time to provision and would not be able to transfer 50 TB within 10 days if the bandwidth is limited to 100 Mbps (or even higher, Snowball is more practical for such large data). Option C (AWS Site-to-Site VPN) is incorrect because it also uses the internet and is subject to the same bandwidth limitations. Option E (AWS DataSync over the internet) is incorrect because it is also limited by the 100 Mbps bandwidth and cannot complete the transfer within 10 days.

Answer

AWS Direct Connect

Answer

AWS Site-to-Site VPN

Answer

AWS DataSync over the internet

Question 11

Refer to the exhibit. A data engineer is creating an IAM policy for an AWS Glue ETL job that reads encrypted objects from an S3 bucket, transforms them, and writes the results back to the same bucket. The bucket uses SSE-KMS encryption with the KMS key specified. The ETL job is failing with an "Access Denied" error when trying to write data. What is the likely cause?

Accepted Answer

The policy is missing the kms:Encrypt permission. The IAM policy must include the kms:Encrypt permission for the AWS Glue ETL job to write encrypted objects to the S3 bucket using SSE-KMS. The policy likely includes s3:PutObject, kms:Decrypt, and kms:GenerateDataKey, but kms:Encrypt is required for the write operation. Options A, B, and C are incorrect because the necessary permissions (kms:Decrypt, s3:PutObjectAcl, and s3:PutObject) are either already present or not required for writing.

Answer

The policy is missing the kms:Decrypt permission

Answer

The policy is missing the s3:PutObjectAcl permission

Answer

The policy is missing the s3:PutObject permission

Question 12

Refer to the exhibit. A data engineer runs the AWS CLI command to check an object in an S3 bucket. The bucket is part of a data lake and is configured with versioning enabled. However, the output shows "VersionId": null. What is the most likely reason for this?

Accepted Answer

The object was uploaded before versioning was enabled. The most likely reason is that the object was uploaded before versioning was enabled on the bucket. When versioning is enabled, objects uploaded afterward receive a unique version ID, while objects that existed before versioning was enabled have a null version ID. Option B is correct. Option A is incorrect because SSE-S3 encryption does not affect version IDs; version IDs are metadata independent of encryption. Option C is incorrect because the `head-object` command automatically returns the version ID of the latest version; no `--version-id` parameter is needed to display it. Option D is incorrect because the bucket is explicitly stated to have versioning enabled.

Answer

The object is encrypted using SSE-S3, which hides the version ID

Answer

The command must include the --version-id parameter to display the version ID

Answer

Versioning is not enabled on the bucket

Question 13

A company runs a streaming data pipeline using Amazon Kinesis Data Streams with 10 shards. The pipeline ingests sensor data from thousands of devices. Each device sends a JSON payload every 5 seconds. The payload size is approximately 2 KB. The data is consumed by a fleet of EC2 instances running a custom Java application that uses the Kinesis Client Library (KCL). Over the past week, the company has observed that the consumer application is experiencing increased latency, and the Kinesis stream's 'GetRecords.IteratorAgeMilliseconds' CloudWatch metric is consistently above 10 seconds. The company has verified that the EC2 instances have sufficient CPU and memory resources. The KCL application is configured with 10 workers, one per shard. The application processes each record by performing a simple transformation and writing to Amazon DynamoDB. The DynamoDB table has sufficient write capacity and is not throttling. The company wants to reduce the iterator age to under 2 seconds. Which action should the company take?

Accepted Answer

Increase the number of shards in the Kinesis stream to 20. The observed high iterator age indicates that the consumer is falling behind. Since the EC2 instances have sufficient resources and DynamoDB is not throttling, the bottleneck is the number of shards. Each shard provides a fixed amount of read throughput (up to 2 MB/s and 5 reads/s). With 10 shards, the consumer's aggregate read throughput is limited. Increasing the number of shards to 20 doubles the read capacity, allowing the KCL workers to fetch records faster and reduce the iterator age to under 2 seconds. Option A is incorrect because Kinesis Data Firehose is a delivery service that buffers and writes to destinations like S3 or Redshift; it does not support custom transformations with a Java application. Option B is incorrect because the DynamoDB table has sufficient write capacity and is not throttling, so increasing it would not address the read-side bottleneck. Option D is incorrect because KCL enforces one worker per shard; adding more workers than shards does not increase parallelism. The workers would sit idle or conflict, since each shard can only be processed by a single worker at a time.

Answer

Replace Kinesis Data Streams with Amazon Kinesis Data Firehose

Answer

Increase the write capacity of the DynamoDB table

Answer

Increase the number of KCL workers to 20

Question 14

A company is streaming e-commerce events to Amazon Kinesis Data Streams. The data science team needs to join events from multiple shards in near real-time and then store the joined results in Amazon S3. Which solution would meet these requirements with the LEAST operational overhead?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to read from the Kinesis stream, perform a join operation using Flink SQL, and write the results to S3 using a sink connector.. Amazon Kinesis Data Analytics for Apache Flink can read from a Kinesis stream, perform stateful joins across shards using Flink SQL or the DataStream API, and write the results to Amazon S3 via a sink connector, all with minimal operational overhead. Option A is wrong because AWS Lambda functions process each shard independently; joining across shards would require managing external state (e.g., DynamoDB), increasing complexity and latency. Option B is wrong because Amazon Kinesis Data Firehose buffers data and writes to S3, but it cannot perform joins; using Athena to join after storage introduces batch-like delays. Option C is wrong because AWS Glue ETL jobs are batch-oriented and not designed for near real-time streaming; Glue Streaming ETL would still require significant configuration and is less optimized for stateful joins across shards.

Answer

Use AWS Lambda functions with Kinesis triggers to process each record, join across shards using a DynamoDB table for state, and write to S3.

Answer

Use Amazon Kinesis Data Firehose to buffer the data and write to S3, then use Amazon Athena to join the data after it is stored.

Answer

Use AWS Glue ETL jobs that read from the Kinesis stream via the Kinesis connector and write the joined results to S3.

Question 15

A data engineer needs to load data from a MySQL database to Amazon S3 daily. The database is 500 GB and the load window is 2 hours. The data must be extracted without impacting the source database performance. Which AWS service should be used to perform the extraction?

Accepted Answer

AWS Database Migration Service (AWS DMS) with a full-load task to S3.. (AWS Database Migration Service). AWS DMS is specifically designed for migrating databases to AWS with minimal impact on the source. It can perform a full-load task to extract data from MySQL and write it to S3 efficiently within the 2-hour window. Option A (AWS Glue ETL) uses JDBC and can cause higher overhead on the source, potentially impacting performance. Option C (Amazon Athena with MySQL federated query) is a query service, not an extraction tool, and may not handle 500 GB efficiently. Option D (Amazon EMR with Spark) is for big data processing and incurs overhead for setup and coordination, making it less suitable for direct daily extraction without impact.

Answer

AWS Glue ETL job using a JDBC connection to read the full table.

Answer

Amazon Athena with the MySQL federated query connector.

Answer

Amazon EMR with a Spark job reading from MySQL via JDBC.