MLS-C01 Data Engineering Practice Test 14 — 15 Questions

Question 1

A data engineering team is building a real-time fraud detection pipeline. The pipeline ingests transaction data from an Amazon Kinesis Data Stream with 10 shards. Each shard produces about 500 records per second, each record is 2 KB. The data is processed by a Lambda function that runs for about 200 ms and then writes results to an Amazon DynamoDB table. The team notices that the Lambda function is experiencing a high number of throttles, and there are increasing numbers of records being retried. The Lambda function's reserved concurrency is set to 100. The DynamoDB table has 100 read capacity units and 100 write capacity units. Which change would most effectively reduce throttling and improve processing throughput?

Accepted Answer

Increase the Lambda function's reserved concurrency to 1000.. The Lambda function is throttling because the concurrent executions needed exceed the reserved concurrency. Each shard invokes Lambda with batches, and with 10 shards and a batch size of 100 (default), the number of concurrent invocations can be high. Increasing reserved concurrency to a higher value (e.g., 1000) allows more concurrent executions, reducing throttling. However, if DynamoDB write capacity is also a bottleneck, increasing it might help. But the primary issue is Lambda throttling.

Answer

Decrease the Lambda function's batch size to 10.

Answer

Increase the DynamoDB write capacity units to 1000.

Answer

Increase the number of shards in the Kinesis stream to 100.

Question 2

A machine learning team is preparing a large dataset for training. The dataset consists of 10,000 CSV files, each about 100 MB, stored in Amazon S3. The team wants to transform the data using AWS Glue ETL jobs. The transformation involves filtering rows, adding new columns, and joining with a small reference table (100 KB). The team is concerned about job performance and cost. They currently have a Glue job with 10 DPU (Data Processing Units) and it takes about 2 hours to complete. The team wants to reduce the runtime and cost. Which approach should they take?

Accepted Answer

Convert the CSV files to Parquet format and partition the data by a column.. Using a columnar format like Parquet and partitioning the data on a relevant column (e.g., date) can significantly reduce the amount of data scanned and improve performance. Additionally, optimizing the number of DPUs (e.g., using a larger number of DPUs for a shorter time) can reduce cost if the job is billed by DPU-hour.

Answer

Use Amazon Athena to transform the data.

Answer

Increase the number of DPUs to 100.

Answer

Use Amazon EMR with Spot Instances instead of AWS Glue.

Question 3

A data engineer is tasked with building a data pipeline that moves data from an on-premises database to Amazon S3 for analytics. The database is a MySQL instance that is 2 TB in size. The company has a 1 Gbps dedicated network connection to AWS (AWS Direct Connect). The data must be transferred once daily. The engineer needs to choose the most efficient and reliable service for this task. Which service should they use?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is designed for migrating databases to AWS and can continuously replicate data. For a one-time daily transfer, DMS can perform a full load and then ongoing replication if needed. It supports MySQL as a source and S3 as a target.

Answer

AWS DataSync

Answer

AWS Glue

Answer

Amazon S3 Transfer Acceleration

Question 4

A data engineering team is using Apache Spark on Amazon EMR to process streaming data from Amazon Kinesis Data Streams. The Spark application uses structured streaming to read from Kinesis, perform transformations, and write to Amazon S3 in Parquet format. The team notices that the application is falling behind and the processing latency is increasing. The Kinesis stream has 5 shards, and the EMR cluster has 5 core nodes of type r5.xlarge. The Spark application is configured with 5 executors, each with 2 cores and 8 GB memory. The team wants to reduce processing latency. Which change would be most effective?

Accepted Answer

Increase the number of shards in the Kinesis stream to 10 and increase the number of core nodes to 10.. The number of shards (5) matches the number of executors (5), but each shard can be processed by a single executor. To increase parallelism, the team should increase the number of shards in the Kinesis stream and correspondingly increase the number of executors or cores. Alternatively, they can increase the number of cores per executor to allow parallel processing of multiple shards per executor.

Answer

Increase the executor memory to 16 GB.

Answer

Use a larger instance type for the core nodes, such as r5.4xlarge.

Answer

Change the output format from Parquet to CSV to reduce write time.

Question 5

A data engineer needs to continuously ingest streaming data from thousands of IoT devices and store the raw data in Amazon S3 for archival processing. The data volume varies significantly throughout the day, and the solution must be serverless, scalable, and cost-effective. Which AWS service should be used to capture and buffer the streaming data before writing to S3?

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose is a serverless service that can capture, transform, and load streaming data into S3. It automatically scales and can buffer data to handle varying volumes. Amazon Kinesis Data Streams (option A) is for real-time processing but requires a consumer to write to S3. Amazon SQS (option C) is a message queue, not designed for streaming data. AWS Glue (option D) is a batch ETL service.

Answer

Amazon Kinesis Data Streams

Answer

AWS Glue

Answer

Amazon Simple Queue Service (SQS)

Question 6

A company is running a machine learning training job on Amazon SageMaker that reads training data from an S3 bucket. The job fails intermittently with an S3 throttling error. The data is partitioned across thousands of small files (average 100 KB). Which strategy is MOST effective to resolve the throttling issue?

Accepted Answer

Combine the small files into larger files (e.g., 100 MB) using a preprocessing step. Combining small files into larger files reduces the number of S3 GET requests, which reduces the chance of throttling. Increasing the number of instances (option A) would increase parallelism and could worsen throttling. Using S3 Transfer Acceleration (option B) improves transfer speed but does not reduce request rate. Using Athena (option D) is for querying, not for training data access.

Answer

Use Amazon Athena to query the data and output results to a new S3 location

Answer

Enable S3 Transfer Acceleration on the bucket

Answer

Increase the number of SageMaker training instances to distribute the load

Question 7

A data scientist wants to explore a large dataset stored in Amazon S3 using SQL queries without moving the data. The dataset is in CSV format and is updated daily with new partitions. Which AWS service should be used to directly query the data in S3?

Accepted Answer

Amazon Athena. Amazon Athena is a serverless interactive query service that allows querying data directly in S3 using standard SQL. Amazon Redshift Spectrum (option B) can also query S3 but requires a Redshift cluster. Amazon EMR (option C) requires cluster management. AWS Glue (option D) is for ETL, not interactive querying.

Answer

Amazon Redshift Spectrum

Answer

Amazon EMR

Answer

AWS Glue

Question 8

A company is building a data pipeline that ingests data from multiple sources into a centralized data lake on Amazon S3. The data must be transformed before it is available for analysis. The pipeline should be event-driven, automatically triggering transformation jobs when new data arrives. Which combination of AWS services should be used?

Accepted Answer

Amazon S3 event notifications to invoke AWS Lambda, which triggers an AWS Glue job. Amazon S3 can send events to AWS Lambda or SQS when new objects are created. AWS Glue can be triggered by Lambda to run ETL jobs. Step Functions (option A) can orchestrate but adds complexity. Kinesis Data Analytics (option B) is for streaming analytics, not batch. EMR (option D) requires cluster management.

Answer

Amazon Kinesis Data Analytics for transformation

Answer

Amazon EMR with automatic scaling

Answer

AWS Step Functions to orchestrate the pipeline

Question 9

A data engineering team is designing a data lake on Amazon S3. They need to enforce encryption at rest for all data stored in the bucket. The security policy requires that the encryption keys be managed by the organization using AWS Key Management Service (KMS), and that the bucket must deny uploads of unencrypted objects. Which bucket policy should be applied?

Accepted Answer

A bucket policy that denies PutObject unless the request includes the 'x-amz-server-side-encryption-aws-kms-key-id' header matching the desired KMS key ID. To enforce encryption, a bucket policy can deny PutObject if the object is not encrypted with the required KMS key. The condition 's3:x-amz-server-side-encryption-aws-kms-key-id' checks the key ID. Option B correctly denies requests that do not include the required key. Option A is incomplete; Option C uses the wrong condition; Option D uses SSE-S3 instead of KMS.

Answer

A bucket policy that denies PutObject unless the request includes the 'x-amz-server-side-encryption' header with value 'AES256'

Answer

A bucket policy that denies PutObject if the 'x-amz-server-side-encryption' header is not present

Answer

Enable default encryption on the bucket with AWS-KMS

Question 10

A company uses Amazon RDS for its transactional database and needs to export a daily snapshot of a table to Amazon S3 in Parquet format for analytics. Which AWS service can perform this export without writing custom code?

Accepted Answer

AWS Glue. AWS Glue can connect to RDS, extract data, and write to S3 in Parquet format using a Glue job. AWS Database Migration Service (DMS) is primarily for migration, not scheduled exports. Amazon Athena cannot directly connect to RDS. Amazon Redshift is a data warehouse, not a migration tool.

Answer

Amazon Redshift

Answer

AWS Database Migration Service (DMS)

Answer

Amazon Athena

Question 11

A company is streaming data from thousands of devices using Amazon Kinesis Data Streams. The data is consumed by a AWS Lambda function that processes each record. The Lambda function is experiencing high error rates and throttling due to the volume of data. Which action would MOST effectively improve the processing throughput and reduce errors?

Accepted Answer

Increase the number of shards in the Kinesis stream. Increasing the number of shards increases the stream's capacity. Using a larger batch size in Lambda (option B) can improve throughput but may cause timeout. Using SQS (option C) introduces another queue. Using Firehose (option D) changes the architecture. Increasing shards is the most direct way to increase throughput.

Answer

Send the data to Amazon SQS first and then process with Lambda

Answer

Use Amazon Kinesis Data Firehose instead of Kinesis Data Streams

Answer

Increase the Lambda function's batch size and reduce the batch window

Question 12

A data scientist needs to run complex ETL transformations on a large dataset stored in Amazon S3. The transformations are written in PySpark and require occasional access to Hive metastore. The solution should minimize operational overhead and allow the data scientist to focus on code development. Which AWS service should be used?

Accepted Answer

Amazon EMR. Amazon EMR is a managed Hadoop framework that supports PySpark and Hive metastore. AWS Glue is good for simpler ETL but has limitations on custom PySpark code. Amazon SageMaker is for ML training, not general ETL. Amazon Redshift is a data warehouse.

Answer

Amazon Redshift

Answer

AWS Glue

Answer

Amazon SageMaker

Question 13

A company wants to perform real-time analytics on streaming data from clickstreams. The data needs to be ingested, processed, and made available for querying within seconds. Which AWS service should be used for the processing step?

Accepted Answer

Amazon Kinesis Data Analytics. Amazon Kinesis Data Analytics is the correct choice because it enables real-time processing and analysis of streaming data using SQL or Apache Flink. It can ingest data from Kinesis Data Streams or Kinesis Data Firehose, process it with sub-second latency, and output results to destinations like Kinesis Data Streams or Firehose for further querying, meeting the requirement for analytics within seconds.

Answer

AWS Glue

Answer

Amazon Redshift

Answer

Amazon Athena

Question 14

A company is using AWS Glue to catalog metadata from various data sources. The crawler is configured to run daily. However, the catalog is not reflecting new partitions added to an S3 bucket during the day. What is the MOST likely cause?

Accepted Answer

The crawler is not scheduled frequently enough to capture changes. Option C is correct because the crawler is configured to run daily, but new partitions are being added to the S3 bucket throughout the day. Since the crawler only runs once per day, it will not detect and catalog those new partitions until its next scheduled run. To capture changes more frequently, the crawler schedule should be increased or an event-driven trigger (e.g., using Amazon S3 Events and AWS Lambda) should be implemented.

Answer

The S3 bucket has insufficient permissions for the Glue crawler

Answer

The table schema has changed and the crawler does not update it

Answer

The data format is not supported by AWS Glue

Question 15

A data engineer is designing a data pipeline that ingests data from a relational database into a data lake on Amazon S3. The data must be incrementally loaded daily. Which TWO AWS services can be used together to achieve this?

Accepted Answer

AWS Glue. AWS Database Migration Service (DMS) can perform continuous replication or scheduled tasks to move data from a relational database to S3. AWS Glue can also connect to databases and run incremental ETL jobs using job bookmarks. Option A (Athena) is for querying, Option B (Kinesis) is for streaming, Option E (Redshift) is a data warehouse.

Answer

Amazon Kinesis Data Streams

Answer

Amazon Athena

Answer

Amazon Redshift