MLS-C01 Data Engineering Practice Test 3 — 15 Questions

Question 1

A company uses AWS Glue ETL jobs to process data from multiple sources. The job fails with the error: 'An error occurred while calling o123.pyWriteDynamicFrame. Insufficient memory.' The job runs on a G.1X worker type with 10 workers. What should be changed to resolve this error?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the number of workers to 20.. The error 'Insufficient memory' in AWS Glue ETL jobs typically indicates that the total memory across all executors is insufficient for the data being processed. Increasing the number of workers from 10 to 20 doubles the total memory and compute capacity available, allowing the job to handle larger datasets without running out of memory. This is the most direct and effective fix for a memory exhaustion error when using the G.1X worker type.

Answer

Enable the Spark UI to monitor the job.

Answer

Change the worker type to G.2X.

Answer

Reduce the number of partitions in the DynamicFrame.

Question 2

A company uses Amazon Redshift as a data warehouse. They need to load 50 TB of clickstream data from S3 into Redshift daily. The data arrives in 5-minute intervals as gzipped CSV files. The target table has a sort key and a distribution key. The load must complete within 2 hours. Which approach is MOST efficient?

Accepted Answer

Use the COPY command with a manifest file and gzip compression.. The COPY command is the most efficient way to load large volumes of data into Amazon Redshift because it uses the cluster's massively parallel processing (MPP) architecture to read data directly from S3 in parallel across all nodes. With a manifest file, you can specify multiple gzipped CSV files, and the gzip compression reduces network I/O and storage overhead. This approach can easily load 50 TB within 2 hours, especially when the target table has a sort key and distribution key, as COPY automatically leverages these for optimal data distribution and sorting during the load.

Answer

Use AWS Glue to transform the data and write to Redshift using JDBC.

Answer

Use a staging table and then merge using a stored procedure.

Answer

Use a series of INSERT statements from a Lambda function.

Question 3

A machine learning engineer needs to process a large dataset that does not fit on a single Amazon SageMaker notebook instance's EBS volume. The data is stored in S3. What is the MOST efficient way to access the data from the notebook?

Accepted Answer

Read the data directly from S3 using the boto3 library.. Reading data directly from S3 using the boto3 library is the most efficient approach for a dataset that exceeds the notebook instance's EBS volume capacity. Boto3 allows you to stream data in chunks or use S3 Select for server-side filtering, avoiding the need to download the entire dataset to local storage. This method leverages S3's high-throughput API and eliminates the bottleneck of writing to a local EBS volume, which is limited in size and I/O performance.

Answer

Increase the EBS volume size to 5 TB.

Answer

Mount the S3 bucket as a file system using s3fs.

Answer

Use SageMaker File input mode in the notebook.

Question 4

A company uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. The application reads from a Kinesis data stream and writes results to a sink. The application is failing with an 'OutOfMemoryError'. The application has parallelism set to 4 and uses 1 Kinesis Processing Unit (KPU). What is the MOST likely cause and solution?

Accepted Answer

The heap memory per operator is too low; increase parallelism to 8.. With parallelism set to 4 but only 1 KPU, each operator slot receives a fraction of the available heap memory, leading to an OutOfMemoryError. Increasing parallelism to 8 distributes the workload across more slots, but more importantly, it forces Kinesis Data Analytics to allocate additional KPUs (each KPU provides 4 GB of memory), thereby increasing the total heap memory available to the application.

Answer

The application is using too many operators; reduce parallelism to 2.

Answer

The checkpoint interval is too short; increase it to 5 minutes.

Answer

The buffer timeout is too high; reduce it to 50 ms.

Question 5

An organization stores sensitive customer data in S3. A data pipeline uses AWS Glue to transform the data and load it into Amazon Redshift. The security team requires that data be encrypted at rest in S3 and in transit between S3 and Glue, and between Glue and Redshift. Which configuration meets these requirements?

Accepted Answer

Enable S3 default encryption with SSE-KMS, use a VPC endpoint for S3, and configure Glue to use SSL for Redshift connection.. It ensures encryption at rest in S3 via SSE-KMS, encrypts data in transit between S3 and Glue by using a VPC endpoint (which enforces HTTPS/TLS), and encrypts data in transit between Glue and Redshift by configuring SSL for the Redshift connection. SSE-KMS provides envelope encryption with a customer-managed key, while the VPC endpoint and SSL satisfy the in-transit encryption requirements.

Answer

Use S3 client-side encryption, and use VPC Peering between Glue and Redshift.

Answer

Use S3 default encryption with SSE-KMS, and use Network Load Balancer for Redshift.

Answer

Enable S3 server-side encryption with SSE-S3, and use SSL for both Glue connections.

Question 6

A data science team is building a real-time fraud detection system. Transactions are streamed via Amazon Kinesis Data Streams, and a Lambda function performs feature engineering and invokes an Amazon SageMaker endpoint for predictions. The team notices that the Lambda function is timing out and causing data loss. Which solution should the team implement to process the stream reliably and at low latency?

Accepted Answer

Use Amazon Kinesis Data Analytics for Apache Flink to consume the stream, perform feature engineering, and invoke the SageMaker endpoint with exactly-once processing.. Amazon Kinesis Data Analytics for Apache Flink provides a stateful, low-latency stream processing engine that can consume from Kinesis Data Streams, perform feature engineering in real-time, and invoke SageMaker endpoints with exactly-once processing semantics. This eliminates Lambda timeouts and data loss by using a long-running, scalable application instead of a short-lived function.

Answer

Use the Kinesis Client Library (KCL) to process the stream in an Amazon EC2 instance, and store the predictions in Amazon DynamoDB.

Answer

Increase the Lambda function timeout to 15 minutes and allocate more memory to reduce processing time.

Answer

Configure Amazon Kinesis Firehose to deliver the stream to an Amazon S3 bucket, then trigger a Lambda function to process the data in batches.

Question 7

A company uses Amazon SageMaker to train and deploy machine learning models. The training data is stored in Amazon S3 (Parquet format, 10 TB). The data scientists have been running training jobs using the File mode input, but the jobs are taking too long due to data download time. They want to reduce the training start-up time and overall training time. Which solution is MOST cost-effective and efficient?

Accepted Answer

Configure the SageMaker training job to use Pipe mode, which streams data directly from S3 without downloading to the instance's local storage.. Pipe mode in SageMaker streams training data directly from Amazon S3 to the training algorithm without first downloading it to the instance's local storage. This eliminates the data download step, significantly reducing startup time and overall training time for large datasets like 10 TB. It is the most cost-effective because it avoids the need for larger instances or additional data transfer acceleration services.

Answer

Use S3 Transfer Acceleration to speed up the data transfer from S3 to the training instance.

Answer

Use larger EC2 instances with more vCPUs and memory to speed up the training process.

Answer

Enable Elastic Fabric Adapter (EFA) on the training instances to improve network throughput.

Question 8

A data engineer is building a data pipeline to process user clickstream data. The data arrives as JSON files in an S3 bucket. The pipeline must transform the JSON into Parquet format and partition by date and event type, then make the data available for Amazon Athena queries. The engineer needs a fully managed, serverless solution with minimal operational overhead. Which combination of AWS services should the engineer use?

Accepted Answer

Use S3 Event Notifications to trigger an AWS Lambda function that converts the JSON to Parquet and writes to a partitioned S3 location, then query with Athena.. AWS Lambda triggered by S3 Event Notifications provides a fully serverless, event-driven architecture with minimal operational overhead for converting JSON to Parquet and partitioning by date and event type. Lambda can process each new JSON file as it arrives, perform the transformation in memory (using libraries like PyArrow or Pandas), and write the Parquet output to a partitioned S3 path, which Athena can then query directly. This approach avoids managing any clusters or job scheduling, aligning with the requirement for a fully managed, serverless solution.

Answer

Use Amazon EMR with Spark to read JSON, convert to Parquet, and partition, then query with Athena.

Answer

Use AWS Glue ETL jobs to read JSON from S3, transform to Parquet, and write to a partitioned S3 location, then use Athena.

Answer

Use Amazon Kinesis Firehose to ingest data and convert to Parquet, then write to S3, and query with Athena.

Question 9

A team is using Amazon SageMaker to train a model on a dataset that is 500 GB in size, stored as CSV files in S3. The training job takes 2 hours using a single ml.p3.2xlarge instance. The team wants to reduce training time to under 30 minutes. The model architecture supports distributed training. Which solution will achieve this goal with the LEAST amount of code changes?

Accepted Answer

Use multiple ml.p3.2xlarge instances with SageMaker's distributed data parallelism library, enabling automatic sharding of the training data.. SageMaker's distributed data parallelism library automatically shards the training data across multiple ml.p3.2xlarge instances, enabling parallel gradient computation and reducing wall-clock training time from 2 hours to under 30 minutes without requiring manual code changes to the training script. The model architecture already supports distributed training, so the library handles the communication and synchronization (e.g., AllReduce) transparently.

Answer

Use managed spot training to reduce cost and then use cost savings to train with a larger instance.

Answer

Use a single ml.p3.16xlarge instance with more GPUs and memory.

Answer

Change the input mode to Pipe mode to stream data from S3 directly, reducing I/O wait time.

Question 10

A company processes large streams of IoT sensor data using Amazon Kinesis Data Streams with 100 shards. Each sensor reading is about 1 KB. The data is consumed by an Amazon EMR cluster running Spark Streaming jobs. The team notices that the Spark Streaming job's processing time is gradually increasing, and the stream is falling behind. They suspect the issue is due to skewed data distribution across shards. Which approach should the team take to diagnose and resolve the issue?

Accepted Answer

Modify the producer to add a random prefix to the partition key, ensuring even distribution across all shards, and monitor the stream using CloudWatch.. Adding a random prefix to the partition key ensures that sensor data is evenly distributed across all 100 shards, eliminating hot shards that cause processing delays. This directly addresses the skewed data distribution issue without requiring infrastructure changes, and the team can monitor the improvement using CloudWatch metrics like IncomingBytes and ReadProvisionedThroughputExceeded.

Answer

Increase the number of shards to 200 to provide more parallelism.

Answer

Check Amazon CloudWatch metrics for Kinesis to identify hot shards, then manually redistribute the data by repartitioning in Spark.

Answer

Use the Kinesis Client Library (KCL) with a custom worker to rebalance the load across shards.

Question 11

A data engineering team is designing a data lake on AWS for machine learning workloads. The data includes structured, semi-structured, and unstructured data. The team needs to ensure that the data is cataloged, easily discoverable, and can be queried by Amazon Athena and Amazon EMR. The team also wants to enforce fine-grained access control at the column and row level for sensitive data. Which combination of AWS services should the team use? (Select TWO.)

Accepted Answer

AWS Lake Formation. AWS Lake Formation is correct because it provides a centralized service to build, secure, and manage data lakes on AWS. It enables fine-grained access control at the column and row level for sensitive data, which directly meets the requirement for enforcing such controls. Additionally, Lake Formation integrates with Amazon Athena and Amazon EMR for querying and processing the cataloged data.

Answer

AWS Identity and Access Management (IAM)

Answer

Amazon RDS for PostgreSQL

Answer

Amazon DynamoDB

Question 12

A company is building a real-time anomaly detection system for network traffic logs. The logs are ingested via Amazon Kinesis Data Streams and processed with an Amazon SageMaker endpoint for inference. The team needs to ensure that the inference results are stored durably and can be replayed for model retraining. The system must handle at least 10,000 records per second with low latency. Which three AWS services should the team use to build this architecture? (Select THREE.)

Accepted Answer

Amazon Kinesis Data Streams. Amazon Kinesis Data Streams is the correct ingestion layer because it provides durable, real-time data streaming with the ability to handle over 10,000 records per second. It acts as the source of truth for network traffic logs, enabling low-latency processing and replay for model retraining.

Answer

AWS Glue ETL

Answer

Amazon Kinesis Data Firehose

Question 13

A data scientist needs to transform raw JSON data from an S3 bucket into Parquet format using AWS Glue. The job must be cost-effective and run only when new data arrives. Which solution should be used?

Accepted Answer

Create a Glue ETL job triggered by an S3 event notification via Lambda.. It uses an S3 event notification to invoke a Lambda function, which then triggers an AWS Glue ETL job only when new data arrives. This event-driven architecture ensures cost-effectiveness by avoiding continuous or scheduled runs, and it directly transforms raw JSON into Parquet format as required.

Answer

Create a Glue crawler that runs continuously.

Answer

Schedule a Glue ETL job to run every hour.

Answer

Use Glue DataBrew to transform data and schedule it daily.

Question 14

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Lambda function that writes to an S3 bucket. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors. What is the MOST likely cause?

Accepted Answer

The Kinesis stream has too few shards for the data volume.. The 'ProvisionedThroughputExceededException' error in Amazon Kinesis Data Streams indicates that the data ingestion rate exceeds the write capacity of the stream's shards. Each shard supports up to 1 MB/s or 1,000 records/s for writes. If the clickstream data volume surpasses this limit, the Lambda function, which reads from the stream, will encounter this exception. Increasing the number of shards scales the write capacity to match the data volume.

Answer

The data retention period of the stream is too short.

Answer

The S3 bucket has insufficient write capacity.

Answer

The Lambda function's reserved concurrency is set too high.

Question 15

A team is building a data pipeline to process terabytes of log data daily using Amazon EMR. The data arrives in 5-minute windows and must be available for querying within 30 minutes. The data is originally in gzip-compressed CSV files. Which approach will minimize processing time and cost?

Accepted Answer

Use Amazon EMR with Spark to convert data to Parquet and store in S3, using spot instances for task nodes.. Converting gzip-compressed CSV to Parquet reduces storage size and improves query performance due to columnar storage and predicate pushdown. Using spot instances for task nodes significantly lowers compute cost, while the 30-minute SLA is achievable with Spark on EMR processing 5-minute windows of data.

Answer

Use Amazon EMR with Spark to convert data to Parquet and use on-demand instances.

Answer

Use AWS Glue to convert data to gzip-compressed CSV and query with Athena.

Answer

Use Amazon EMR with Hive to transform data to compressed CSV and store in S3.