Knowledge + Practice

CCNA Data Ingestion and Transformation Questions

75 of 591 questions · Page 6/8 · Data Ingestion and Transformation · Answers revealed

Practice these questions Domain overview All questions

376

MCQmedium

A financial company needs to ingest real-time stock trade data from multiple sources and store it in Amazon S3 for compliance. The data must be delivered within 1 minute of the trade occurring. The data volume is approximately 10,000 records per second, with occasional spikes to 50,000 records per second. The engineer has set up Amazon Kinesis Data Streams with 10 shards and a Kinesis Data Firehose delivery stream that reads from the Kinesis stream and writes to S3. However, during spikes, the Firehose delivery stream falls behind, causing data to be delayed beyond the 1-minute SLA. What should the engineer do to meet the SLA without over-provisioning?

A.Increase the buffer size in Kinesis Data Firehose from 1 MB to 5 MB to batch more data per delivery.

B.Use Amazon SQS as a buffer between Kinesis and Firehose to absorb spikes.

C.Replace Firehose with an AWS Lambda function that writes directly to S3 for lower latency.

D.Increase the number of shards in the Kinesis data stream to handle peak throughput and enable auto-scaling.

AnswerD

More shards increase read capacity; auto-scaling adjusts during spikes.

Why this answer

Increasing the number of shards in the Kinesis data stream allows it to handle the peak throughput of 50,000 records per second, and enabling auto-scaling ensures the stream adapts to varying loads without manual intervention. This reduces the backlog that causes Firehose to fall behind, meeting the 1-minute SLA. Option A is incorrect because increasing the buffer size in Firehose would increase latency, not reduce it.

Option B is incorrect because adding SQS as a buffer adds another hop, increasing latency and complexity. Option C is incorrect because AWS Lambda may not scale to 50,000 records per second and adds processing latency.

Practice this question →

377

Multi-Selectmedium

A company uses Amazon Kinesis Data Streams to ingest clickstream data. The data must be transformed and stored in Amazon S3 for batch analytics. The engineer wants to use AWS Lambda for transformation. Which TWO configurations are required? (Choose two.)

Select 2 answers

A.Configure the Lambda function to write to S3 via Kinesis Data Firehose.

B.Configure the Kinesis stream to send records directly to S3.

C.Set up an SQS queue as a destination for Lambda errors.

D.Create an event source mapping from the Kinesis stream to the Lambda function.

E.Assign an IAM role to Lambda with permissions to read from Kinesis and write to S3.

AnswersD, E

Event source mapping enables Lambda to poll from Kinesis.

Why this answer

The correct answers are D and E. D: An event source mapping is required for Lambda to poll records from Kinesis. E: Lambda needs an IAM role with permissions to read from Kinesis and write to S3.

Option A is incorrect because Lambda does not write to S3 via Firehose; Firehose is a separate service that could directly write to S3, but in this scenario, Lambda is the transformation actor and writes directly to S3. Option B is incorrect because a Kinesis stream cannot send records directly to S3; it needs a consumer or Firehose. Option C is incorrect because Lambda errors can be sent to SQS but it is not required for this setup; the question asks for required configurations.

Practice this question →

378

MCQmedium

A company needs to ingest real-time sensor data from thousands of IoT devices into Amazon S3, with a latency of less than 1 minute. The data must be transformed (e.g., convert to Parquet) before landing in S3. Which combination of services is MOST cost-effective?

A.Amazon Kinesis Data Streams to AWS Lambda to S3.

B.Amazon Kinesis Data Streams to Amazon Kinesis Data Analytics to S3.

C.Amazon Kinesis Data Streams to AWS Glue streaming ETL to S3.

D.Amazon Kinesis Data Streams to Amazon Kinesis Data Firehose to S3.

AnswerD

Firehose can transform and deliver with low latency, cost-effective for high throughput.

Why this answer

The most cost-effective because Amazon Kinesis Data Firehose directly ingests data from Kinesis Data Streams, can perform transformations (e.g., convert to Parquet) using built-in or Lambda functions, and delivers to S3 with low latency (under 1 minute). Option A is incorrect because AWS Lambda, while capable of transformation, does not scale cost-effectively for thousands of devices due to per-invocation costs and concurrency limits. Option B is incorrect because Kinesis Data Analytics is designed for complex stream processing with SQL, not simple transformations, and adds unnecessary cost.

Option C is incorrect because AWS Glue streaming ETL is more suited for near-real-time batch processing and can incur higher costs and latency for high-throughput ingestion.

Practice this question →

379

Multi-Selecthard

A company uses a Kinesis Data Firehose delivery stream to load data into an S3 bucket. The data is in JSON format and must be converted to Parquet before landing in S3. Which steps are required to achieve this? (Choose THREE.)

Select 3 answers

A.Configure the Firehose delivery stream to enable data format conversion to Parquet.

B.Create a table in the AWS Glue Data Catalog with the schema.

C.Store the schema in Amazon DynamoDB.

D.Set the Firehose's schema mapping to reference the Glue table.

E.Use Kinesis Data Analytics to convert the data.

AnswersA, B, D

Firehose has built-in conversion capability.

Why this answer

Kinesis Data Firehose natively supports converting incoming data from JSON to Parquet format. This conversion is enabled directly in the delivery stream configuration, eliminating the need for separate processing steps.

Exam trap

The trap here is that candidates may think DynamoDB is needed for schema storage or that Kinesis Data Analytics is required for the conversion, but Firehose's built-in Parquet conversion with Glue schema support is the correct and simpler approach.

Practice this question →

380

MCQhard

A data engineer runs an AWS Glue ETL job that writes to a table in the AWS Glue Data Catalog. The job fails occasionally with the error 'Resource Not Found' for the table. The table exists. What is a likely cause?

A.The job is using an outdated version of the table schema.

B.Multiple Glue jobs are writing to the same table concurrently.

C.The table location in S3 is incorrect.

D.The Glue job name contains special characters.

AnswerA

Schema version mismatch can cause 'Resource Not Found'.

Why this answer

AWS Glue jobs can cache the table schema at job start. If the table schema is updated while the job is running, the job may still reference the old schema version, which may no longer exist in the Data Catalog, causing a 'Resource Not Found' error even though the table exists. Option B is incorrect because concurrent writes do not cause this specific error; they may cause conflicts but not a 'Resource Not Found' for the table itself.

Option C is incorrect because if the table exists and the location is wrong, it would typically result in a different error (e.g., 'Access Denied' or 'PathNotFound') rather than 'Resource Not Found' for the table. Option D is incorrect because special characters in the job name do not affect table access.

Practice this question →

381

MCQmedium

A company uses Amazon S3 to store raw data and needs to transform it into Parquet format for analytics. The transformation job runs daily on a schedule. Which AWS service is BEST suited for this task?

A.Amazon Redshift

B.Amazon EMR

C.AWS Lambda

D.AWS Glue

AnswerD

Glue is serverless, supports Parquet, and can be scheduled with triggers.

Why this answer

AWS Glue is a fully managed, serverless ETL service that can automatically convert data formats (e.g., from CSV to Parquet) and run on a schedule (e.g., daily). It is ideal for this use case because it is purpose-built for ETL transformations and handles schema discovery, data cataloging, and job scheduling without managing infrastructure. Option A (Amazon Redshift) is wrong because Redshift is a data warehouse for querying, not a transformation service; it could load Parquet but not convert raw data to Parquet directly.

Option B (Amazon EMR) is wrong because EMR requires provisioning and managing clusters, adding operational overhead. Option C (AWS Lambda) is wrong because Lambda has a maximum execution timeout of 15 minutes, which is too short for daily large-scale data transformation jobs.

Practice this question →

382

MCQmedium

A data engineer runs an AWS Glue job that reads from a JDBC connection to a PostgreSQL database. The job fails with a 'Connection timed out' error. The Glue job runs in a VPC with the appropriate security group. What is the most likely cause?

A.The network ACL associated with the Glue job's subnet is blocking outbound traffic.

B.The Glue job does not have permission to access the database.

C.The security group does not allow inbound traffic from the Glue job.

D.The database credentials are incorrect.

AnswerA

Network ACLs can block traffic.

Why this answer

The 'Connection timed out' error indicates a network connectivity issue. Since the Glue job runs in a VPC with a security group that likely allows outbound traffic, the most likely cause is that the network ACL (which is stateless) associated with the Glue job's subnet is blocking outbound traffic to the database. Option A is correct.

Option B is incorrect because the error is not an authentication or permission issue. Option C is incorrect because the security group's inbound rule does not affect outbound traffic from the Glue job. Option D is incorrect because the error is network-related, not a credentials issue.

Practice this question →

383

Multi-Selecthard

Which THREE factors should be considered when choosing between AWS Glue and Amazon EMR for data transformation? (Choose three.)

Select 3 answers

A.Glue automatically stores data in S3 after transformation.

B.EMR allows fine-grained control over cluster configuration and software.

C.EMR supports real-time stream processing with Spark Streaming.

D.Glue is serverless, reducing operational overhead.

E.Glue integrates natively with the Glue Data Catalog for schema management.

AnswersB, D, E

EMR provides flexibility to install custom software and tune clusters.

Why this answer

Amazon EMR provides full control over cluster configuration, including the ability to customize software, install libraries, and tune Spark, Hadoop, or Hive parameters. This fine-grained control is essential for complex or specialized data transformation pipelines that require specific versions or custom configurations.

Exam trap

The trap here is that candidates may confuse Glue's automatic schema discovery with automatic data storage, or assume EMR is the only option for streaming, when in fact both services support streaming but with different levels of control and operational overhead.

Practice this question →

384

MCQmedium

A company uses Amazon Kinesis Data Streams to ingest clickstream data from a website. The data is consumed by an AWS Lambda function that writes to Amazon DynamoDB. The Lambda function is seeing high error rates due to DynamoDB write throttling. Which action should be taken to reduce throttling?

A.Use Amazon Kinesis Data Firehose instead of Kinesis Data Streams

B.Add an Amazon SQS queue between Lambda and DynamoDB

C.Increase the Lambda function memory

D.Enable auto scaling on the DynamoDB table

AnswerD

Auto scaling adjusts write capacity to handle spikes and reduce throttling.

Why this answer

Enabling DynamoDB auto scaling increases write capacity automatically when needed. Using Kinesis Data Firehose would change the architecture but does not address throttling directly. Increasing Lambda memory does not help with DynamoDB throttling.

Using SQS would add a queue but does not increase DynamoDB capacity.

Practice this question →

385

Multi-Selecthard

A data engineer is troubleshooting a slow-running AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job processes 500 GB of CSV data daily. The engineer wants to improve performance. Which THREE actions should the engineer take? (Choose three.)

Select 3 answers

A.Use a JDBC connection with a higher batch size for writing to Redshift.

B.Partition the input data in S3 by date or category.

C.Switch to a single-node Redshift cluster to reduce latency.

D.Increase the number of DPUs allocated to the Glue job.

E.Reduce the number of input files by combining them into larger files.

AnswersA, B, D

Larger batch sizes reduce round trips and improve write throughput.

Why this answer

Increasing the JDBC batch size for the Redshift connection reduces the number of network round trips and improves write throughput. The Glue JDBC connector batches rows into a single INSERT statement; a larger batch size (e.g., 1000 instead of the default 100) allows more rows per commit, reducing overhead and speeding up the write phase.

Exam trap

The trap here is that candidates often assume combining files always improves performance (due to Hadoop's small file problem), but in Glue ETL with Spark, moderate parallelism from many files is beneficial, and the real bottleneck is often the JDBC write path, not the S3 read path.

Practice this question →

386

MCQeasy

A company is using AWS Glue to run ETL jobs that transform data from Amazon DynamoDB to Amazon S3. The DynamoDB table has a large number of items (over 10 million) and is heavily used by production applications. The Glue job reads the entire DynamoDB table each time it runs, causing increased read capacity consumption and affecting production performance. The team wants to reduce the impact on the source DynamoDB table while still keeping the S3 data up-to-date. What should the team do?

A.Use DynamoDB Streams and AWS Lambda to capture changes and write them to S3, then run incremental Glue jobs.

B.Increase the DynamoDB read capacity units to handle the Glue job's read load.

C.Use the DynamoDB console to export the table to S3 in Parquet format.

D.Reduce the parallelism of the Glue job to lower the read throughput.

AnswerA

Captures only changes, reducing read impact.

Why this answer

Using DynamoDB Streams with AWS Lambda enables incremental change data capture (CDC), which eliminates the need to read the entire DynamoDB table each time. This reduces read capacity consumption and minimizes impact on production performance. Option B is incorrect because increasing read capacity units would still involve full table scans, further straining the production workload.

Option C is incorrect because exporting via the DynamoDB console is a one-time export, not an incremental solution to keep S3 data up-to-date. Option D is incorrect because reducing Glue job parallelism does not change the fact that the entire table is read, and it would increase job duration without addressing the read capacity issue.

Practice this question →

387

Multi-Selectmedium

A data engineer is designing a near-real-time streaming pipeline to ingest clickstream data from a web application. The data must be enriched with user metadata from a DynamoDB table before being stored in S3. Which combination of AWS services should the engineer use? (Choose TWO.)

Select 2 answers

A.Amazon Kinesis Data Firehose

B.Amazon Kinesis Data Analytics for Apache Flink

C.Amazon Kinesis Data Streams

D.AWS Lambda with DynamoDB Accelerator (DAX)

E.AWS Glue Streaming ETL

AnswersB, C

Performs stream enrichment with DynamoDB lookups.

Why this answer

Amazon Kinesis Data Streams (C) provides the low-latency, durable ingestion layer for the clickstream data, while Amazon Kinesis Data Analytics for Apache Flink (B) allows you to run a Flink application that can perform stream-to-stream joins with the DynamoDB user metadata in near-real time. The enriched output can then be written to S3 via a Kinesis Data Firehose delivery stream or directly from the Flink application.

Exam trap

The trap here is that candidates often confuse Kinesis Data Firehose's ability to invoke a Lambda for simple transformations with the need for stateful stream enrichment, leading them to select Firehose alone without a stream processing engine.

Practice this question →

388

MCQeasy

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that ingests JSON log data from web servers. The stream is configured to transform records with an AWS Lambda function and deliver to an Amazon S3 bucket. Recently, the stream has been failing with 'InvalidData' errors. Which action should the engineer take to resolve the issue?

A.Verify the S3 bucket policy allows Firehose to write.

B.Increase the buffer size and interval in the Firehose delivery stream.

C.Change the data format to CSV in the Firehose configuration.

D.Check the CloudWatch Logs for the Lambda function to identify transformation errors.

AnswerD

Lambda errors are logged in CloudWatch and can reveal why transformation fails.

Why this answer

The 'InvalidData' error in Kinesis Data Firehose typically indicates that the Lambda function used for data transformation is failing or returning malformed records. By checking the CloudWatch Logs for the Lambda function, the engineer can identify specific transformation errors, such as incorrect JSON parsing, missing fields, or exceptions, which cause Firehose to reject the records. This is the most direct troubleshooting step because Firehose relies on the Lambda function to return valid transformed data in the expected format.

Exam trap

The trap here is that candidates often confuse 'InvalidData' errors with S3 permission issues or buffer configuration problems, but the error specifically points to a failure in the data transformation step, not the delivery destination or batching settings.

How to eliminate wrong answers

Option A is wrong because if the S3 bucket policy were the issue, the error would be a permission or access denied error, not 'InvalidData'. Option B is wrong because increasing buffer size or interval would not resolve data transformation errors; it only affects how data is batched before delivery. Option C is wrong because changing the data format to CSV would not fix transformation errors; Firehose expects the Lambda function to return data in the same format as the input (JSON) unless explicitly configured otherwise, and the 'InvalidData' error is unrelated to the output format.

Practice this question →

389

Multi-Selectmedium

A company is designing a data ingestion pipeline for real-time sensor data from thousands of devices. The data must be processed with low latency and stored in Amazon S3. Which TWO services would be appropriate for this use case? (Choose TWO.)

Select 2 answers

A.AWS Glue

B.AWS DataSync

C.Amazon Athena

D.Amazon Kinesis Data Firehose

E.Amazon Kinesis Data Streams

AnswersD, E

Firehose can deliver streaming data to S3 with buffering.

Why this answer

Amazon Kinesis Data Firehose is the correct choice because it is a fully managed service designed to ingest real-time streaming data, transform it on the fly (e.g., convert to Parquet/ORC), and deliver it directly to Amazon S3 with low latency. It handles buffering, compression, and partitioning automatically, making it ideal for the described sensor data pipeline.

Exam trap

The trap here is that candidates may choose only one streaming service, but the question requires two services, and the correct pairing is Kinesis Data Streams for real-time ingestion and Kinesis Data Firehose for delivery to S3, as Firehose alone cannot provide sub-second latency.

Practice this question →

390

Multi-Selecthard

A data engineer is designing a data ingestion pipeline that uses AWS DMS to migrate data from an on-premises Oracle database to Amazon S3 in Parquet format. The engineer needs to ensure that data is continuously replicated with minimal latency. Which THREE steps should the engineer take? (Choose three.)

Select 3 answers

A.Configure a DMS task with a transformation rule to convert to Parquet.

B.Specify an S3 bucket as the target endpoint with data format set to Parquet.

C.Use AWS Schema Conversion Tool (SCT) to convert the schema.

D.Enable change data capture (CDC) on the source database.

E.Perform a full load only, without CDC.

AnswersA, B, D

Correct. A transformation rule in the DMS task converts the data to Parquet format during migration.

Why this answer

Options A, B, and D are correct. To continuously replicate data with minimal latency from Oracle to Amazon S3 in Parquet format, the engineer should: (A) configure a DMS task with a transformation rule to convert to Parquet, (B) specify an S3 bucket as the target endpoint with data format set to Parquet, and (D) enable change data capture (CDC) on the source database. Option C (using AWS SCT) is incorrect because SCT is for schema conversion, not for DMS replication.

Option E (full load only) is incorrect because it does not provide continuous replication.

Practice this question →

391

MCQmedium

A data engineer needs to design a data ingestion pipeline that ingests CSV files from an Amazon S3 bucket, transforms the data by adding a timestamp column, and loads it into an Amazon Redshift table. The pipeline should run automatically whenever a new file is uploaded to the S3 bucket. Which AWS service should be used to trigger the transformation?

A.AWS Lambda

B.AWS Step Functions

C.Amazon EventBridge

D.Amazon Simple Queue Service (SQS)

AnswerA

Lambda can be triggered directly by S3 events.

Why this answer

Amazon S3 can be configured to send events directly to AWS Lambda when a new CSV file is uploaded. Lambda then executes the transformation (adding a timestamp column) and loads the data into Redshift. Option B (Step Functions) is not triggered directly by S3 events without an intermediate service like Lambda.

Option C (EventBridge) can route S3 events to Lambda, but a direct S3 event notification to Lambda is simpler and more common; however, the key point is that Lambda is the direct trigger. Option D (SQS) requires a separate process to poll the queue and invoke Lambda; it is not a direct trigger.

Practice this question →

392

MCQhard

A company needs to transfer 50 TB of historical data from an on-premises HDFS cluster to Amazon S3. The network bandwidth is limited to 1 Gbps, and the transfer must complete within 10 days. The data is compressible. Which solution is MOST appropriate?

A.Use AWS DataSync over a Direct Connect connection.

B.Use Amazon S3 Transfer Acceleration with multipart uploads.

C.Use multiple parallel AWS CLI sync commands over the internet.

D.Use AWS Snowball Edge to physically ship the data.

AnswerA

DataSync can saturate 1 Gbps, but 50 TB at 1 Gbps takes ~111 hours, plus overhead may exceed 10 days.

Why this answer

With a 1 Gbps link, transferring 50 TB would take approximately 111 hours (about 4.6 days) theoretically, well within the 10-day window. AWS DataSync over a Direct Connect connection provides a high-speed, secure, and reliable method to transfer large datasets online. DataSync automates the HDFS-to-S3 transfer, handles compression, and can saturate the 1 Gbps link, making it the most appropriate solution.

Snowball Edge involves physical shipment and logistics, which would likely exceed the 10-day deadline. Options B and C also rely on internet bandwidth and are less efficient than DataSync.

Practice this question →

393

MCQhard

The exhibit shows an IAM policy attached to a role used by an AWS Glue ETL job. The job reads from an S3 bucket and writes to another S3 bucket. However, the job fails with an access denied error when trying to write to the output bucket. What is the most likely cause?

A.The policy is missing permissions for AWS KMS to decrypt/encrypt objects

B.The policy does not allow glue:StartJobRun on the specific job

C.The policy only allows PutObject on the my-data-lake bucket, but the job writes to a different bucket

D.The policy does not include s3:ListBucket permission

AnswerC

The S3 permissions are scoped to my-data-lake/*; if output bucket is different, access is denied.

Why this answer

The policy allows s3:PutObject on my-data-lake/*, but if the output bucket is different (e.g., my-output-bucket), the policy does not cover it. The error is due to missing permissions on the output bucket. The Glue service role may not have permissions to write to the output bucket.

The policy does not restrict resource to only one bucket, but the ARN specifies my-data-lake. The job might be trying to write to a different bucket. There is no issue with Glue actions.

Practice this question →

394

Multi-Selecteasy

A company is using AWS Glue to catalog data in Amazon S3. The data is in CSV format with varying schemas. The Data Engineering team wants to ensure the Glue Data Catalog is updated automatically when new partitions are added to S3. Which TWO actions should be taken? (Choose two.)

Select 2 answers

A.Enable partition indexing on the Glue Data Catalog.

B.Set up an S3 event notification to trigger a Lambda function that updates the Glue Data Catalog.

C.Configure a scheduled AWS Glue crawler to run on a regular basis.

D.Run Amazon Athena queries with MSCK REPAIR TABLE to add partitions.

E.Use AWS Glue ETL jobs to write data and update the catalog simultaneously.

AnswersA, C

Partition indexing enables automatic updates and efficient querying of new partitions.

Why this answer

A is correct because enabling partition indexing in the Glue Data Catalog allows partition pruning and automatic updates. C is correct because configuring a Glue crawler with a schedule will automatically discover new partitions. B is wrong because setting up an S3 event notification to trigger Lambda for manual updates is not as efficient as crawler scheduling.

D is wrong because using AWS Glue ETL jobs to update the catalog is not automatic. E is wrong because Amazon Athena does not update the catalog.

Practice this question →

395

MCQeasy

A company needs to ingest data from a self-managed Apache Kafka cluster running on EC2 into Amazon S3. The data must be delivered in near real-time. Which AWS service is BEST suited for this task?

A.Use Amazon MSK to replicate the Kafka cluster and then use a connector to S3.

B.Use Amazon S3 Transfer Acceleration to speed up the transfer from Kafka brokers to S3.

C.Use Amazon Kinesis Data Streams as an intermediary to buffer data before writing to S3.

D.Use an AWS Glue streaming ETL job that reads from the Kafka cluster and writes to S3.

AnswerD

Glue supports streaming from Kafka and can write to S3.

Why this answer

AWS Glue streaming ETL jobs can connect directly to an Apache Kafka cluster (including self-managed) as a source and write data to Amazon S3 in near real-time, making it a fully managed and suitable solution. Option A (Amazon MSK) is a managed Kafka service but does not directly ingest into S3; Option B (S3 Transfer Acceleration) only accelerates uploads, not ingestion; Option C (Kinesis Data Streams) adds unnecessary complexity as an intermediary.

Practice this question →

396

MCQhard

A company runs an AWS Glue ETL job that reads data from Amazon S3, transforms it, and writes back to S3 in a different partition structure. The job uses the 'spark.sql.shuffle.partitions' option set to 200. After the job completes, the output has many small files. The data engineer wants to minimize the number of output files while maintaining job performance. Which action should the engineer take?

A.Use 'coalesce(n)' with n based on target file size (e.g., 128 MB) before writing.

B.Enable S3 multipart upload for the Glue job.

C.Increase the 'spark.sql.shuffle.partitions' to 500.

D.Reduce the 'spark.sql.shuffle.partitions' to 50.

AnswerA

Coalesce reduces partitions without a full shuffle, minimizing files.

Why this answer

'coalesce(n)' reduces the number of partitions without triggering a full shuffle, allowing you to control the number of output files based on a target file size (e.g., 128 MB). This minimizes small files while preserving job performance, as coalesce is a narrow transformation that avoids the overhead of a shuffle. In contrast, 'repartition(n)' would cause a full shuffle, degrading performance.

Exam trap

The trap here is that candidates often confuse 'coalesce' with 'repartition' or assume that adjusting 'spark.sql.shuffle.partitions' directly controls output file count, when in fact it only controls the number of partitions during shuffle operations, not the final write partition count.

How to eliminate wrong answers

Option B is wrong because enabling S3 multipart upload does not reduce the number of output files; it only improves upload reliability and throughput for large objects, but the job still writes the same number of small files. Option C is wrong because increasing 'spark.sql.shuffle.partitions' to 500 would increase the number of shuffle partitions, leading to even more small output files and potentially worse performance due to higher task overhead. Option D is wrong because reducing 'spark.sql.shuffle.partitions' to 50 would reduce the number of shuffle partitions, but it does not directly control the number of output files written; it may cause data skew and memory pressure, and the output file count still depends on the final partition count, which may remain high if the job uses repartition or other transformations.

Practice this question →

397

Multi-Selecthard

A data engineer is troubleshooting an AWS Glue job that reads from Amazon RDS MySQL and writes to Amazon S3. The job runs successfully but takes longer than expected. The engineer wants to optimize performance. Which THREE actions would improve job performance?

Select 3 answers

A.Increase the number of DPUs allocated to the Glue job.

B.Use a single JDBC connection per partition.

C.Increase the JDBC fetch size parameter.

D.Convert the output format from Parquet to CSV.

E.Use a pushdown predicate to filter data at the source.

AnswersA, C, E

More DPUs provide more parallelism.

Why this answer

Increasing the number of DPUs (Data Processing Units) allocated to the AWS Glue job provides more parallel processing capacity, allowing the job to process data faster. This is a direct way to improve performance when the job is CPU or memory-bound, as Glue distributes the workload across the allocated DPUs.

Exam trap

The trap here is that candidates might think converting to CSV improves performance due to simplicity, but in reality, Parquet's columnar storage and compression provide significant performance benefits for analytics workloads on S3.

Practice this question →

398

MCQhard

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data includes a timestamp field. They want to partition the S3 objects by hour dynamically. The Firehose delivery stream is configured with a prefix like 'year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/'. However, the objects are not being partitioned as expected; all files end up in a single partition. What is the MOST likely cause?

A.Dynamic partitioning is not enabled in the Firehose delivery stream configuration.

B.The buffer size is set too large, delaying file delivery.

C.The timestamp is in UTC but the prefix uses local time.

D.The IAM role for Firehose lacks permissions to write to S3 with dynamic prefixes.

AnswerA

Without enabling dynamic partitioning, the prefix is static and all data goes into one S3 prefix.

Why this answer

Amazon Kinesis Data Firehose requires dynamic partitioning to be explicitly enabled in the delivery stream configuration in order to use custom partitioning keys and create partition prefixes dynamically. Without enabling dynamic partitioning, the timestamp expressions in the prefix are treated as static strings, resulting in a single partition. Option B is incorrect because buffer size affects delivery frequency, not partitioning behavior; even with large buffers, partitioning would still occur if dynamic partitioning were enabled.

Option C is incorrect because time zone differences would cause incorrect hours, not a single partition; all files would still be distributed across hours. Option D is incorrect because IAM permission issues would cause write failures, not mispartitioning; the objects would simply not be delivered.

Practice this question →

399

Multi-Selectmedium

Which TWO AWS services can be used to ingest streaming data from a mobile application into Amazon S3 for near-real-time analytics? (Choose 2.)

Select 2 answers

A.Amazon Kinesis Data Firehose

B.Amazon Kinesis Data Streams

C.Amazon DynamoDB Streams

D.AWS Glue

E.Amazon SQS

AnswersA, B

Firehose can ingest streaming data and deliver to S3 near real-time.

Why this answer

Amazon Kinesis Data Firehose is the correct choice because it is a fully managed service designed to load streaming data directly into Amazon S3, Redshift, Elasticsearch, or Splunk without requiring custom code. It can capture and transform streaming data from mobile applications in near-real-time and automatically deliver it to S3, making it ideal for near-real-time analytics pipelines.

Exam trap

The DEA-C01 exam often tests the distinction between managed ingestion (Firehose) and raw stream processing (Data Streams), and the trap here is that candidates may incorrectly choose DynamoDB Streams or SQS because they associate 'streaming' with any service containing 'stream' or 'queue', without understanding that DynamoDB Streams only captures internal table changes and SQS requires custom code to write to S3.

Practice this question →

400

MCQeasy

An organization uses AWS Lake Formation to manage a data lake in S3. A new data engineer needs to create a Glue ETL job that reads from a Lake Formation-managed table. The engineer has been granted SELECT permission on the table via Lake Formation. However, the job fails with an AccessDenied error. What is the MOST likely cause?

A.The IAM role used by the Glue job does not have Lake Formation permissions.

B.The S3 bucket policy does not allow the Glue job to access the data.

C.The table has not been registered with Lake Formation.

D.The Glue job is not running in the same VPC as Lake Formation.

AnswerA

Correct. The IAM role must have Lake Formation permissions to access the table.

Why this answer

The IAM role used by the Glue job must have Lake Formation permissions (such as lakeformation:GetDataAccess) to access the table. Without these permissions, the job will fail with an AccessDenied error even if the S3 bucket policy allows access. Option B is incorrect: S3 bucket policies are not the primary issue because Lake Formation manages fine-grained access; the role must have Lake Formation permissions.

Option C is incorrect: the table is already registered with Lake Formation if it is a Lake Formation-managed table; registration is a prerequisite for granting permissions. Option D is incorrect: Lake Formation does not require a specific VPC; Glue jobs can access Lake Formation over the internet or via a VPC endpoint, but that is not the cause of the AccessDenied error.

Practice this question →

401

Multi-Selecteasy

A data engineer needs to transform data in an S3 data lake using AWS Glue ETL. The data is in CSV format and needs to be converted to Parquet with partitioning by date. The engineer wants to minimize the number of files written to S3 to improve query performance. Which TWO configuration options should the engineer use? (Select TWO.)

Select 2 answers

A.Increase the number of workers in the Glue job to increase parallelism.

B.Use the coalesce method to reduce the number of output partitions.

C.Disable compression in the Parquet output.

D.Enable partition pruning in the Glue job by setting the 'partitionKeys' parameter.

E.Set the 'groupFiles' option to 'inPartition' in the DynamicFrame writer.

AnswersB, D

Coalesce reduces the number of partitions before writing, resulting in fewer files.

Why this answer

Using `coalesce` reduces the number of output partitions, which directly minimizes the number of files written to S3. Fewer, larger Parquet files improve query performance by reducing the overhead of file listing and metadata operations in engines like Amazon Athena or Redshift Spectrum.

Exam trap

The trap here is that candidates often confuse increasing parallelism (Option A) with improving performance, but in this context, more parallelism leads to more small files, which degrades query performance; the correct approach is to reduce file count via coalesce and enable partition pruning.

Practice this question →

402

MCQmedium

A company is ingesting log files from EC2 instances into CloudWatch Logs and then wants to deliver them to S3 for long-term storage and analysis. The data engineer needs to ensure the logs are delivered to S3 within 5 minutes of being generated. Which approach meets this requirement?

A.Configure a CloudWatch Logs metric filter and invoke a Lambda function to write to S3

B.Use CloudWatch Logs Insights to query logs and save results to S3

C.Use CloudWatch Logs subscription filter with Kinesis Data Firehose to deliver to S3

D.Use the CloudWatch Logs export to S3 feature

AnswerC

Firehose can deliver to S3 within minutes.

Why this answer

CloudWatch Logs subscription filters can stream log data in near real-time to Kinesis Data Firehose, which then delivers the data to S3 with a buffer interval configurable down to 60 seconds, easily meeting the 5-minute requirement. This approach provides the lowest latency for automated, continuous delivery without custom code.

Exam trap

The trap here is that candidates often confuse the batch export feature (which has a 12-hour latency) with a real-time solution, or assume a Lambda-based approach is simpler without considering the latency and management overhead of custom code.

How to eliminate wrong answers

Option A is wrong because CloudWatch Logs metric filters are designed to extract metric data from logs, not to trigger Lambda functions for each log event; while you could use a subscription filter with Lambda, the metric filter itself cannot invoke Lambda, and Lambda-based delivery would add latency and complexity. Option B is wrong because CloudWatch Logs Insights is an interactive query engine for ad-hoc analysis, not a mechanism for automated, continuous delivery to S3; it requires manual intervention to save results. Option D is wrong because the CloudWatch Logs export to S3 feature is a batch export operation that can take up to 12 hours to complete, far exceeding the 5-minute requirement.

Practice this question →

403

MCQhard

A financial services company ingests real-time stock trade data using Amazon Kinesis Data Streams with 10 shards. Each shard receives about 500 records per second, each record approximately 1 KB. The data is consumed by a single AWS Lambda function that transforms the data and writes to Amazon S3. The Lambda function is configured with 1024 MB memory and a timeout of 5 minutes. The company notices that the Lambda function is frequently throttled, and data ingestion lags behind. The Lambda function's CloudWatch metrics show that the iterator age is increasing, and the function's concurrency is maxed out at 1000. The data engineer needs to resolve the throttling issue without changing the Lambda function code. What should the data engineer do?

A.Increase the number of shards in the Kinesis data stream to increase parallelism.

B.Reduce the Lambda function memory to 512 MB to increase concurrency limit.

C.Decrease the batch size to 10 records to reduce processing time per invocation.

D.Increase the Lambda function memory to 2048 MB to improve processing speed.

AnswerA

More shards allow more Lambda concurrent executions, reducing iterator age.

Why this answer

Increasing the number of shards increases the number of Kinesis Data Streams processing units, which directly increases the concurrency limit for Lambda consumers. With more shards, more Lambda function instances can process records in parallel, reducing the iterator age and alleviating throttling. Option B is wrong because reducing memory would likely degrade performance and not increase concurrency limit (concurrency limit is independent of memory).

Option C is wrong because decreasing batch size increases the number of invocations, potentially worsening throttling and overhead. Option D is wrong because increasing memory may improve per-record processing speed, but the core issue is concurrency limit being maxed out; increasing memory does not increase concurrency limit and may not resolve throttling if CPU is not the bottleneck.

Practice this question →

404

MCQhard

A data engineer is ingesting XML data from an external API into Amazon S3. The engineer needs to transform the XML to JSON using AWS Glue. The XML structure is deeply nested. Which Apache Spark method should be used in the Glue ETL script?

A.Use the built-in AWS Glue 'xml' data source

B.Use Hadoop's XmlInputFormat

C.Use spark.read.format('xml') with Databricks XML library

D.Use the Spark SQL function from_xml()

AnswerC

This is the standard way to parse XML in Spark.

Why this answer

The Databricks XML library (spark.read.format('xml')) provides native support for parsing deeply nested XML into a DataFrame, which is essential for AWS Glue ETL scripts running on Spark. AWS Glue does not have a built-in 'xml' data source, and the Databricks library handles complex nested structures, attributes, and arrays automatically, making it the standard approach for XML-to-JSON transformation in Spark.

Exam trap

The DEA-C01 exam often tests the misconception that AWS Glue has a built-in 'xml' data source (Option A), when in fact Glue relies on external Spark libraries like Databricks XML for XML processing, and candidates confuse the Spark SQL function from_xml() with a file-level reader.

How to eliminate wrong answers

Option A is wrong because AWS Glue does not have a built-in 'xml' data source; Glue's native formats are JSON, Parquet, ORC, Avro, and CSV, and using 'xml' would throw an error. Option B is wrong because Hadoop's XmlInputFormat is designed for MapReduce jobs, not Spark DataFrames, and it outputs key-value pairs of XML fragments, requiring manual parsing and lacking schema inference for deeply nested XML. Option D is wrong because from_xml() is a Spark SQL function that parses a single string column containing XML into struct columns, but it cannot read an entire XML file or dataset; it requires the XML to already be loaded as a string, making it unsuitable for ingesting raw XML files from S3.

Practice this question →

405

MCQeasy

Refer to the exhibit. An S3 event notification is configured to trigger an AWS Lambda function when objects are created in 'my-bucket'. The Lambda function processes the JSON file and writes results to Amazon DynamoDB. The function fails with a timeout error. Which action should the engineer take to resolve the issue?

A.Modify the S3 event notification to use a different event type

B.Grant the Lambda function permission to access DynamoDB

C.Change the trigger to Amazon SQS instead of S3

D.Increase the Lambda function timeout

AnswerD

Timeout error indicates the function needs more time.

Why this answer

The Lambda function is failing with a timeout error, which indicates that the function is taking longer to execute than the default timeout of 3 seconds. Increasing the Lambda function timeout allows the function to run longer and complete its processing of the JSON file and DynamoDB write operation without being prematurely terminated.

Exam trap

The DEA-C01 exam often tests the distinction between timeout errors and permission errors, leading candidates to incorrectly choose a permissions fix (Option B) when the error message explicitly states 'timeout'.

How to eliminate wrong answers

Option A is wrong because changing the S3 event notification to a different event type (e.g., from s3:ObjectCreated:* to s3:ObjectCreated:Put) does not address the timeout issue; the function still fails due to execution duration, not the trigger event. Option B is wrong because a timeout error is not a permissions issue; if the Lambda function lacked DynamoDB permissions, it would fail with an access denied error (e.g., 403), not a timeout. Option C is wrong because changing the trigger to Amazon SQS instead of S3 does not resolve the timeout; the Lambda function would still have the same execution duration limit and would timeout regardless of the trigger source.

Practice this question →

406

MCQmedium

Refer to the exhibit. A data engineer created this IAM policy for a Lambda function that reads from a Kinesis stream and writes to an S3 bucket. The Lambda function fails with an 'AccessDenied' error when trying to write to S3. What is the missing permission?

A.s3:ListBucket on the bucket

B.s3:GetObject on the bucket

C.s3:PutObjectAcl on the bucket

D.s3:DeleteObject on the bucket

AnswerC

If the bucket policy requires object ACLs, s3:PutObjectAcl may be necessary alongside PutObject. This is the most plausible missing permission from the given options.

Why this answer

The IAM policy includes s3:PutObject, which is sufficient for writing objects to S3. The AccessDenied error indicates the bucket policy or the resource ARN in the policy is misconfigured. Among the given options, s3:PutObjectAcl might be required if the bucket is configured to require ACLs on write operations.

Practice this question →

407

MCQeasy

A data engineer needs to ingest JSON files from an S3 bucket into a DynamoDB table. The files are updated hourly and contain new records. Which AWS service should be used to trigger a Lambda function for each new object?

A.Kinesis Data Firehose

B.Amazon EventBridge

C.S3 Event Notifications

D.Amazon SQS

AnswerC

S3 can send events to Lambda on object creation.

Why this answer

S3 Event Notifications are the correct choice because they are designed to trigger AWS Lambda functions directly in response to object creation events (e.g., `s3:ObjectCreated:*`) in an S3 bucket. This allows the data engineer to automatically invoke a Lambda function for each new JSON file as it is uploaded, enabling ingestion into DynamoDB without polling or additional infrastructure.

Exam trap

The trap here is that candidates may confuse Amazon EventBridge with S3 Event Notifications, but EventBridge is not the native S3 event trigger—S3 Event Notifications are the direct, service-integrated mechanism for invoking Lambda on object creation.

How to eliminate wrong answers

Option A is wrong because Kinesis Data Firehose is a streaming ingestion service that loads data into destinations like S3, Redshift, or Elasticsearch, but it cannot directly trigger a Lambda function for each new object; it is not an event source for Lambda. Option B is wrong because Amazon EventBridge can schedule or react to events from various AWS services, but it is not the native, direct integration for S3 object creation events—S3 Event Notifications are the simpler and more appropriate mechanism. Option D is wrong because Amazon SQS is a message queue service that decouples components, but it does not natively trigger Lambda from S3 events without additional configuration (e.g., S3 Event Notification to SQS, then Lambda polling SQS), which adds unnecessary complexity for this use case.

Practice this question →

408

MCQeasy

A data engineer is troubleshooting an AWS Glue ETL job that fails with the error: 'An error occurred while calling o137.pyWriteDynamicFrame. No such file or directory: s3://bucket/output/part-00000.parquet'. The job reads from a JDBC source and writes to S3. What is the most likely cause?

A.The schema of the source data has changed, causing a mismatch during write

B.The output S3 path does not exist and the Glue job does not have permission to create it

C.The Glue job ran out of memory during the transformation phase

D.The IAM role attached to the Glue job lacks permissions to read from the JDBC source

AnswerB

The error message indicates missing directory; Glue may not auto-create if permissions are insufficient.

Why this answer

The error 'No such file or directory: s3://bucket/output/part-00000.parquet' indicates that the Glue job is trying to write to an S3 path that does not exist. By default, AWS Glue does not automatically create the target S3 bucket or prefix; it requires the path to already exist or the IAM role to have s3:PutObject permissions that allow the S3 service to create the object. Since the error occurs at the write stage (pyWriteDynamicFrame) and not during read, the most likely cause is that the output S3 path does not exist and the Glue job lacks the necessary permissions to create it.

Exam trap

The trap here is that candidates often confuse a missing S3 path with a permissions issue, but the error message explicitly states 'No such file or directory', which points to the path not existing rather than a generic access denied, and the exam expects you to recognize that Glue does not auto-create the output S3 prefix.

How to eliminate wrong answers

Option A is wrong because a schema mismatch during write would typically cause a different error, such as a schema compatibility or type conversion error, not a 'No such file or directory' file system error. Option C is wrong because an out-of-memory error would manifest as a Java heap space or memory limit exception, not a missing file error on S3. Option D is wrong because the error occurs during the write phase (pyWriteDynamicFrame) and not during the JDBC read; if the IAM role lacked JDBC read permissions, the job would fail earlier with a connection or authentication error.

Practice this question →

409

MCQhard

A company uses AWS Glue to transform data in Amazon S3. The transformation logic is written in Python and references several libraries that are not included in the default Glue environment. Which approach should the data engineer use to make these libraries available?

A.Include a requirements.txt file in the Glue job script and run pip install during job initialization.

B.Package the libraries in an AWS Lambda layer and attach it to the Glue job.

C.Upload the libraries as a .zip file to an S3 bucket and reference them in the Glue job's Python library path.

D.Use the --additional-python-modules parameter in the Glue job.

AnswerC

Glue Python shell jobs allow adding custom Python modules from S3.

Why this answer

AWS Glue allows you to provide custom Python libraries by uploading them as a .zip file to an S3 bucket and referencing the S3 path in the 'Python library path' field of the job. This method is supported for both Glue ETL and Python shell jobs, making custom libraries available at runtime. Option A is incorrect because Glue does not execute arbitrary shell commands like 'pip install' during job initialization; the job script runs in a controlled environment.

Option B is incorrect because AWS Lambda layers are specific to Lambda functions and cannot be attached to Glue jobs. Option D is incorrect because the --additional-python-modules parameter, while valid in Glue 3.0 and later, only supports installing packages from PyPI, not custom libraries not published on PyPI.

Practice this question →

410

MCQmedium

A company wants to ingest streaming data from thousands of IoT devices into AWS for real-time analytics. The data volume is variable and can spike unpredictably. The solution must be serverless and minimize operational overhead. Which AWS service should be used for ingestion?

A.Use Amazon Kinesis Data Firehose to load streaming data directly into Amazon S3.

B.Use Amazon SQS to queue messages and process them in batches.

C.Use Amazon Kinesis Data Streams to ingest and process data in real time.

D.Use AWS IoT Core to ingest data and route it to Amazon DynamoDB.

AnswerC

Amazon Kinesis Data Streams is a serverless streaming data service that can handle variable and high-throughput data from many sources, making it ideal for IoT data ingestion.

Why this answer

Amazon Kinesis Data Streams is the correct choice because it is a serverless, real-time data ingestion service designed to handle variable and unpredictable data volumes from thousands of sources. It provides durable, low-latency streaming with the ability to process data in real time using consumers like Lambda or Kinesis Data Analytics, meeting the requirements for real-time analytics and minimal operational overhead.

Exam trap

The trap here is that candidates often confuse Kinesis Data Firehose (near-real-time, batch delivery) with Kinesis Data Streams (real-time, sub-second processing), assuming Firehose is sufficient for real-time analytics when it actually introduces latency due to its buffering and batching behavior.

How to eliminate wrong answers

Option A is wrong because Amazon Kinesis Data Firehose is a near-real-time delivery service that batches data before loading it into destinations like S3, which introduces latency and does not support sub-second real-time analytics; it also lacks the ability to process data in real time without additional services. Option B is wrong because Amazon SQS is a message queue service designed for decoupling and asynchronous processing, not for real-time streaming ingestion; it does not support ordered, replayable, or high-throughput streaming from thousands of IoT devices, and batch processing adds latency. Option D is wrong because AWS IoT Core is a managed IoT platform that can ingest data from devices but is primarily for device management and MQTT/HTTP communication, not optimized for high-volume, variable streaming data ingestion for real-time analytics; routing to DynamoDB is a specific use case and does not provide the flexible, scalable streaming ingestion needed for real-time analytics.

Practice this question →

411

Multi-Selecteasy

A data engineer needs to ingest data from a SaaS application (Salesforce) into Amazon S3 on a daily basis. Which TWO AWS services can be used for this purpose? (Choose TWO.)

Select 2 answers

A.AWS DataSync

B.Amazon Kinesis Data Streams

C.AWS Transfer Family

D.AWS Glue

E.Amazon AppFlow

AnswersD, E

Glue can connect to Salesforce via JDBC and write to S3.

Why this answer

AWS Glue is correct because it provides a fully managed ETL service that can connect to Salesforce via JDBC and extract data into Amazon S3 on a scheduled basis. Glue's crawlers and jobs can handle incremental loads and schema evolution, making it suitable for daily ingestion from SaaS sources.

Exam trap

The trap here is that candidates may confuse AWS DataSync's ability to sync from cloud sources (like EFS) with SaaS applications, but DataSync does not support Salesforce or other SaaS APIs natively.

Practice this question →

412

MCQhard

A company is using AWS Lake Formation to manage permissions on data in Amazon S3. They need to ingest data from an external source into a new database 'sales_db' and a table 'transactions' using AWS Glue. The IAM role used by Glue must have the minimal permissions to create the database and table in the Data Catalog and write data to the S3 location. Which combination of permissions should be granted?

A.IAM policy with `glue:CreateDatabase`, `glue:CreateTable`, and `s3:PutObject`

B.IAM policy with `lakeformation:GrantPermissions` on the database and table

C.IAM policy with `s3:GetObject` and `s3:PutObject` on the target location

D.Lake Formation permissions: `CREATE_DATABASE` on the catalog, `CREATE_TABLE` on `sales_db`, and S3 location permission

AnswerD

Lake Formation controls Data Catalog operations; S3 write is also needed.

Why this answer

AWS Lake Formation manages permissions on the Data Catalog. To create a database and table, the Glue IAM role must have Lake Formation `CREATE_DATABASE` on the catalog and `CREATE_TABLE` on the `sales_db` database. Additionally, to write data to the S3 location, the role needs S3 write permissions (either via an IAM policy or Lake Formation data location permissions).

Option A lacks Lake Formation permissions. Option B grants overly broad `lakeformation:GrantPermissions` which is not needed for creation. Option C provides only S3 permissions, missing Data Catalog permissions.

Thus, D is the minimal combination.

Practice this question →

413

MCQhard

Refer to the exhibit. A data engineer has configured an S3 event notification to send an event to an SQS queue when objects are created in the 'incoming/' prefix. The engineer wants to trigger an AWS Lambda function to process the object. However, the Lambda function is not being invoked. What is the most likely cause?

A.The Lambda function lacks permission to read from the S3 bucket.

B.Lambda is not configured as an event source for the SQS queue.

C.The SQS queue does not exist or is in a different account.

D.The S3 event notification filter prefix is incorrect.

AnswerB

Lambda must poll the SQS queue to be triggered.

Why this answer

The Lambda function is not being invoked because the SQS queue is not configured as an event source for Lambda. Even though S3 sends events to SQS, Lambda will only poll and process messages from the queue if an event source mapping (e.g., via CreateEventSourceMapping) is established. Without this mapping, the messages sit in the queue and Lambda remains idle.

Exam trap

The DEA-C01 exam often tests the distinction between sending events to a queue (S3 → SQS) and actually consuming them (SQS → Lambda), so candidates mistakenly assume that simply having S3 send to SQS will automatically trigger Lambda without an explicit event source mapping.

How to eliminate wrong answers

Option A is wrong because the Lambda function does not need permission to read from the S3 bucket directly; it only needs permission to read from the SQS queue (via the event source mapping) and to access the object in S3 once triggered. Option C is wrong because if the SQS queue did not exist or was in a different account, the S3 event notification would fail immediately (S3 would log an error), but the question states the event is sent to SQS, implying the queue exists and is accessible. Option D is wrong because the S3 event notification filter prefix 'incoming/' is correctly configured to match objects created under that prefix; if it were incorrect, no events would be sent to SQS at all.

Practice this question →

414

MCQhard

A company uses AWS DMS to migrate an on-premises Oracle database to Amazon RDS for PostgreSQL. The migration completes, but the target table has more rows than the source. Which is the MOST likely cause?

A.DMS used binary replication which included extra metadata rows.

B.Oracle and PostgreSQL handle case sensitivity differently.

C.DMS performed a full load instead of ongoing replication.

D.Target tables lack unique constraints, causing DMS to insert duplicate rows.

AnswerD

Without unique constraints, DMS may re-apply changes and create duplicates.

Why this answer

When target tables lack unique constraints (like primary keys or unique indexes), AWS DMS cannot identify duplicate records during change data capture (CDC). If the same change is applied multiple times (e.g., due to recovery after a failure), DMS may insert duplicate rows, causing the target to have more rows than the source. Option A is incorrect because binary replication is not used; DMS uses logical replication.

Option B is incorrect because case sensitivity differences could lead to missing rows, not extra rows. Option C is incorrect because a full load alone would not cause duplicates; the issue arises during ongoing replication without unique constraints.

Practice this question →

415

MCQhard

A company is using Amazon Kinesis Data Streams with a Lambda consumer to process real-time events. The Lambda function is triggered by a DynamoDB stream to update a counter. Recently, the counter has been inaccurate due to duplicate processing. What is the most likely cause?

A.The DynamoDB stream is configured with 'TRIM_HORIZON' iterator type

B.The Lambda function's reserved concurrency is too low

C.The Lambda function is not idempotent and is being retried on failures

D.The Kinesis stream has undergone a shard rebalance

AnswerC

Retries cause duplicate updates if the function is not idempotent.

Why this answer

The Lambda function is invoked at least once per record in the DynamoDB stream. If the function fails or times out, it may be retried, processing the same record again. If the function is not idempotent, this retry leads to duplicate updates to the counter.

Option A is incorrect because TRIM_HORIZON is a Kinesis iterator type, not relevant to DynamoDB streams. Option B is incorrect because low reserved concurrency would cause throttling, not duplicate processing. Option D is incorrect because shard rebalancing occurs in Kinesis, not DynamoDB streams, and does not cause duplicates.

Practice this question →

416

MCQhard

A data engineer runs a weekly AWS Glue ETL job that processes data from Amazon DynamoDB to Amazon S3. The job reads the entire table every time, which is slow and expensive. The job needs to process only items that changed since the last run. Which solution should the engineer implement?

A.Use DynamoDB Scan with a LastEvaluatedKey to paginate and store the last scanned key to resume next time

B.Enable DynamoDB Streams and process change events with AWS Lambda to write to S3

C.Add a Global Secondary Index (GSI) on a timestamp attribute and query only new records

D.Use AWS Database Migration Service (DMS) with ongoing replication from DynamoDB to S3

AnswerB

Streams capture item-level changes, enabling incremental loads.

Why this answer

DynamoDB Streams captures item-level changes (inserts, updates, deletes) in near real-time. An AWS Lambda function can process these change events and write only the incremental data to Amazon S3, eliminating the need to scan the entire DynamoDB table. This approach is both cost-effective and efficient for incremental data ingestion.

Exam trap

The trap here is that candidates may think a GSI on a timestamp (Option C) is sufficient for incremental processing, but it fails to capture updates to existing items that do not change the timestamp, and it still requires a full scan of the index to find new records.

How to eliminate wrong answers

Option A is wrong because using DynamoDB Scan with LastEvaluatedKey still reads the entire table every time; it only paginates the results, not reduces the data read. Option C is wrong because a Global Secondary Index (GSI) on a timestamp attribute does not automatically track changes; you would still need to query for new records based on a timestamp, which requires storing the last processed timestamp and can miss updates to existing items. Option D is wrong because AWS Database Migration Service (DMS) with ongoing replication is designed for continuous database migration and can be complex to set up for simple incremental loads; it is overkill compared to using DynamoDB Streams with Lambda, and DMS does not natively write to S3 in a format optimized for analytics without additional transformation.

Practice this question →

417

MCQmedium

A data engineer needs to transform JSON data into Parquet format using AWS Glue. The input data has nested fields. Which Glue feature should be used to flatten the nested structure?

A.Relationalize transform

B.DropNullFields transform

C.FindMatches transform

D.Map transform

AnswerA

Relationalize transforms nested JSON into flat tables.

Why this answer

The Relationalize transform is the correct choice because it is specifically designed to flatten nested JSON structures (such as arrays and structs) into a set of related tables (DataFrames) that can be written as Parquet. This transform recursively extracts nested fields, creating separate DataFrames for each level of nesting, which is essential for converting complex JSON into a flat, columnar Parquet format suitable for analytics.

Exam trap

The trap here is that candidates may confuse the Map transform (which can flatten JSON with custom code) with a built-in flattening feature, but AWS Glue's Relationalize is the dedicated, no-code solution for this specific task, and the exam expects you to know the exact purpose of each transform.

How to eliminate wrong answers

Option B (DropNullFields transform) is wrong because it only removes fields with null values from the schema, not flattening nested structures. Option C (FindMatches transform) is wrong because it is used for fuzzy matching and deduplication of records, not for schema transformation. Option D (Map transform) is wrong because it applies a custom function to each row or column but does not inherently flatten nested JSON; it requires manual coding to handle nesting, whereas Relationalize automates the process.

Practice this question →

418

MCQeasy

A data engineer needs to schedule an AWS Glue ETL job to run every hour and process new data that arrives in an S3 bucket. The job should only process files that have been added since the last run. Which approach should the engineer use to track which files have been processed?

A.Configure S3 Event Notifications to trigger the Glue job on each new object creation.

B.Enable job bookmarks in the Glue ETL job.

C.Store the last processed timestamp in a DynamoDB table and query it at the start of the job.

D.Use S3 Inventory to list all objects and filter by last modified date in the job.

AnswerB

Glue job bookmarks automatically track the state of data processed and only process new data.

Why this answer

AWS Glue job bookmarks automatically track the last processed data, allowing the job to incrementally process only new files since the last run. This is the native, built-in mechanism for stateful incremental processing in Glue ETL, eliminating the need for external tracking.

Exam trap

The trap here is that candidates often choose Option A (S3 Event Notifications) because it seems like a direct trigger for new files, but they overlook that the requirement is for an hourly batch job that tracks processed files across runs, not a per-file event-driven trigger.

How to eliminate wrong answers

Option A is wrong because S3 Event Notifications trigger the job per object creation, which can lead to duplicate processing if multiple files arrive within the hour and does not inherently track which files have been processed across runs. Option C is wrong because storing the last processed timestamp in DynamoDB requires custom implementation and maintenance, and it does not handle edge cases like file overwrites or partitions as reliably as Glue bookmarks. Option D is wrong because S3 Inventory provides periodic snapshots (daily or weekly) and is not designed for real-time or hourly incremental processing; filtering by last modified date in the job would require full scans and manual state management.

Practice this question →

419

MCQeasy

A company wants to schedule a nightly batch job to copy data from an on-premises PostgreSQL database to Amazon S3. The solution must minimize operational overhead. Which AWS service should be used?

A.AWS Glue

B.AWS Data Pipeline

C.Amazon EMR

D.AWS Database Migration Service (AWS DMS) with ongoing replication

AnswerA

AWS Glue can run scheduled ETL jobs to read from PostgreSQL and write to S3 with minimal overhead.

Why this answer

AWS Glue is the correct choice because it provides a fully managed ETL service that can connect to on-premises PostgreSQL via JDBC, extract data, and write it to Amazon S3 with minimal configuration. Glue's built-in scheduler can run the job nightly, eliminating the need to manage servers or orchestration infrastructure, which directly meets the requirement to minimize operational overhead.

Exam trap

The trap here is that candidates often confuse AWS DMS (designed for continuous replication) with batch data movement, overlooking that DMS's ongoing replication feature is not intended for scheduled batch jobs and adds unnecessary overhead for a simple nightly copy.

How to eliminate wrong answers

Option B (AWS Data Pipeline) is wrong because, while it can copy data from on-premises databases to S3, it requires managing a task runner on-premises and has higher operational overhead compared to Glue's serverless model. Option C (Amazon EMR) is wrong because it is designed for big data processing using Hadoop/Spark clusters and introduces significant overhead for cluster management, making it overkill for a simple nightly batch copy job. Option D (AWS DMS with ongoing replication) is wrong because it is primarily intended for database migration and continuous replication, not for scheduled batch jobs; using it for nightly batch copies would incur unnecessary complexity and cost for ongoing change data capture.

Practice this question →

420

Multi-Selecthard

A data engineer is troubleshooting an AWS Glue job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but 5% of records are missing after the load. The engineer suspects data consistency issues. Which THREE actions could help diagnose and resolve the problem? (Choose THREE.)

Select 3 answers

A.Use the Redshift COPY command with a manifest file to load data.

B.Increase the number of DPUs for the Glue job.

C.Enable Glue job bookmarks to track processed files.

D.Use a staging table in Redshift with a transaction to commit.

E.Review the job's CloudWatch Logs for any error messages.

AnswersA, C, E

Manifest file ensures all files are loaded.

Why this answer

Using the Redshift COPY command with a manifest file ensures that only the exact files listed in the manifest are loaded, eliminating the risk of partial or duplicate reads from S3. This is a common pattern to guarantee data consistency when the Glue job may not reliably track which files have been processed, especially in scenarios with concurrent writes or retries.

Exam trap

The trap here is that candidates often assume performance tuning (increasing DPUs) or database-level transactions (staging tables) can fix data ingestion gaps, when the actual problem is incomplete or inconsistent file discovery from the source (S3).

Practice this question →

421

Multi-Selecteasy

A data engineer is designing a data ingestion pipeline for streaming social media data. The data must be ingested with low latency (seconds) and stored in Amazon S3 for long-term analytics. The engineer also needs to perform real-time aggregations. Which TWO services should the engineer use? (Choose two.)

Select 2 answers

A.Amazon Kinesis Data Firehose

B.AWS Glue ETL

C.Amazon S3

D.Amazon Kinesis Data Analytics

E.Amazon Kinesis Data Streams

AnswersD, E

Performs real-time analytics on streaming data.

Why this answer

Amazon Kinesis Data Analytics is correct because it enables real-time processing and aggregation of streaming data using SQL or Apache Flink, allowing the engineer to compute metrics like counts or averages on the fly. Combined with Kinesis Data Streams as the ingestion layer, it provides sub-second latency for both ingestion and analytics, meeting the low-latency requirement.

Exam trap

The trap here is that candidates often confuse Amazon Kinesis Data Firehose with Kinesis Data Streams, assuming Firehose can achieve sub-second latency, but Firehose's minimum buffer interval of 60 seconds makes it unsuitable for true real-time ingestion, while Data Streams provides the necessary low-latency ingestion.

Practice this question →

422

MCQmedium

A data engineer is using AWS Glue ETL to transform a large dataset in S3. The job processes 2 TB of data daily and currently runs for 6 hours. The engineer wants to reduce runtime without changing the transformation logic. What is the best approach?

A.Reduce the number of DPUs to minimize overhead.

B.Use the Spark UI to analyze bottlenecks and rewrite code.

C.Increase the number of Glue DPUs or enable auto-scaling.

D.Switch from Spark to Python shell.

AnswerC

More DPUs provide parallel processing and reduce runtime.

Why this answer

Increasing the number of DPUs or enabling auto-scaling directly allocates more distributed processing capacity to the AWS Glue job, which reduces runtime for large datasets by parallelizing the workload across more resources. Since the transformation logic is fixed and the job is already running on Spark, adding compute capacity is the most straightforward way to speed up processing without code changes.

Exam trap

The trap here is that candidates may think reducing DPUs reduces overhead and speeds up the job, but in distributed systems, more parallelism (more DPUs) reduces runtime for large datasets, while reducing DPUs increases it.

How to eliminate wrong answers

Option A is wrong because reducing DPUs would decrease parallelism and likely increase runtime, not reduce it, as the job already takes 6 hours on 2 TB of data. Option B is wrong because using the Spark UI to analyze bottlenecks and rewriting code would change the transformation logic, which the question explicitly prohibits. Option D is wrong because switching from Spark to Python shell would remove distributed processing entirely, making the job unable to handle 2 TB of data efficiently and likely causing it to fail or run far longer.

Practice this question →

423

MCQeasy

A data engineer needs to ingest data from an on-premises Oracle database into Amazon S3. The data volume is about 500 GB initially, with daily incremental updates of 10 GB. The pipeline must minimize operational overhead. Which AWS service should be used for the initial and incremental loads?

A.AWS Database Migration Service (DMS) with change data capture (CDC) to Amazon S3.

B.AWS Glue with a JDBC connection and incremental crawl.

C.Amazon Kinesis Data Firehose with a custom producer.

D.AWS Data Pipeline with a SQL activity and HiveCopyActivity.

AnswerA

DMS supports full load and CDC with low overhead.

Why this answer

AWS DMS with CDC is the correct choice because it supports continuous replication from Oracle to Amazon S3 with minimal overhead. It handles both the initial 500 GB full load and ongoing 10 GB daily increments via change data capture, without requiring custom code or complex pipeline management.

Exam trap

The trap here is that candidates often choose AWS Glue for its serverless nature, but Glue's incremental crawl only updates the Data Catalog, not the data itself, and it cannot capture row-level changes from a database without full reloads.

How to eliminate wrong answers

Option B is wrong because AWS Glue with an incremental crawl is designed for cataloging schema changes, not for capturing row-level changes from a database; it would require full table scans for each incremental load, which is inefficient for 10 GB daily updates. Option C is wrong because Amazon Kinesis Data Firehose requires a custom producer to stream data from Oracle, which adds operational overhead and does not natively support CDC or initial bulk loads from a database. Option D is wrong because AWS Data Pipeline with a SQL activity and HiveCopyActivity is a legacy service that lacks native CDC support for Oracle, requiring custom scripting for incremental loads and increasing operational complexity.

Practice this question →

424

MCQeasy

A company needs to transform JSON data from Amazon Kinesis Data Streams into Parquet format and store it in Amazon S3. The transformation includes simple field mappings and type conversions. Which approach is most cost-effective and serverless?

A.Use Amazon EC2 instances running Apache Spark Streaming

B.Use Amazon Kinesis Data Firehose with an AWS Lambda function for transformation and output to Parquet

C.Use Amazon SageMaker for transformation

D.Use an AWS Glue ETL job triggered by a Kinesis stream

AnswerB

Firehose can invoke Lambda per record and convert to Parquet.

Why this answer

Amazon Kinesis Data Firehose can directly deliver streaming data to Amazon S3 in Parquet format. By attaching an AWS Lambda function for simple field mappings and type conversions, you achieve a fully serverless, cost-effective solution without managing any infrastructure. This approach minimizes costs because you pay only for the data volume processed by Firehose and the Lambda invocations, avoiding the overhead of always-on compute resources.

Exam trap

The trap here is that candidates often overestimate the need for full ETL engines like Glue or Spark for simple transformations, overlooking that Kinesis Data Firehose with Lambda is the most cost-effective and serverless option for lightweight field mappings and format conversions.

How to eliminate wrong answers

Option A is wrong because using Amazon EC2 instances running Apache Spark Streaming requires provisioning and managing servers, incurring continuous compute costs even when idle, and is not serverless. Option C is wrong because Amazon SageMaker is designed for machine learning model training and inference, not for lightweight streaming data transformations like field mappings and type conversions. Option D is wrong because an AWS Glue ETL job triggered by a Kinesis stream introduces additional complexity and cost from Glue's Spark-based processing, which is overkill for simple transformations and is not as cost-effective as Firehose with Lambda.

Practice this question →

425

Multi-Selecthard

Which THREE factors should be considered when choosing between Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose for real-time data ingestion? (Choose three.)

Select 3 answers

A.The need for a fully managed delivery destination

B.Whether the application requires custom data processing logic

C.The ability to compress data before storage

D.The latency requirements for data delivery

E.The maximum throughput supported per shard

AnswersA, B, D

Firehose can directly deliver to S3, Redshift, etc., while Streams requires a consumer.

Why this answer

Amazon Kinesis Data Firehose is a fully managed service that automatically delivers streaming data to destinations like Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk, making it ideal when you need a fully managed delivery destination without managing the ingestion pipeline. In contrast, Amazon Kinesis Data Streams requires you to build and manage consumers to process and deliver data, so if you need a fully managed destination, Firehose is the correct choice.

Exam trap

The trap here is that candidates often assume compression or throughput limits are unique to one service, but both services support compression and throughput is a scaling detail of Streams, not a direct comparison factor for choosing between the two.

Practice this question →

426

MCQhard

A data engineer is building a real-time data pipeline using Amazon Kinesis Data Streams with a Lambda consumer. The data volume is 2 MB/s with average record size of 5 KB. The Lambda function processes records and writes to DynamoDB. Occasionally, the Lambda function fails with 'ProvisionedThroughputExceededException' on DynamoDB. What is the best way to handle this?

A.Replace Lambda with the Kinesis Client Library (KCL) running on EC2.

B.Increase the Lambda function's reserved concurrency to process more records in parallel.

C.Use DynamoDB Streams to capture the records and process them asynchronously.

D.Configure a Lambda destination on failure to send records to an SQS dead-letter queue, and implement retry logic in Lambda.

AnswerD

This handles transient failures without data loss.

Why this answer

Configuring a Lambda destination on failure to send unprocessed records to an SQS dead-letter queue, combined with retry logic in the Lambda function, provides a robust mechanism to handle DynamoDB throttling exceptions. This approach decouples the retry handling from the Kinesis stream, preventing the Lambda function from blocking the shard iterator and allowing the pipeline to continue processing other records while failed records are retried or investigated.

Exam trap

The trap here is that candidates often assume increasing concurrency or switching to KCL will solve throughput issues, but the real problem is handling DynamoDB throttling gracefully without blocking the Kinesis stream, which requires a decoupled retry mechanism like a dead-letter queue.

How to eliminate wrong answers

Option A is wrong because replacing Lambda with KCL on EC2 does not inherently solve DynamoDB throttling; it shifts compute but still requires managing write capacity and retry logic, adding operational overhead without addressing the root cause. Option B is wrong because increasing Lambda reserved concurrency would increase the number of concurrent invocations, potentially exacerbating DynamoDB throttling by sending more write requests simultaneously, not reducing failures. Option C is wrong because DynamoDB Streams capture changes after writes to DynamoDB, not before; they cannot help handle write failures from the Lambda function, as the exception occurs during the write attempt, not after.

Practice this question →

427

MCQhard

A company uses AWS Glue to run ETL jobs that process data from Amazon S3 and load into Amazon Redshift. The jobs have recently started failing with 'Out of Memory' errors. The data volume has increased 3x in the past month. Which is the MOST effective solution to resolve this issue without redesigning the job?

A.Use Amazon Athena instead of Glue for the transformation.

B.Increase the number of Glue workers (DPUs) for the job.

C.Rewrite the job to use Spark SQL instead of PySpark.

D.Increase the number of partitions in the input S3 data.

AnswerB

More workers provide more memory and CPU to handle increased data volume.

Why this answer

To increase the number of Glue workers (DPUs). This provides more memory and processing capacity to handle the increased data volume, directly resolving the 'Out of Memory' errors. Increasing S3 partitions (option D) may improve parallelism but does not directly increase memory for the Glue job.

Using Spark SQL (option C) instead of PySpark does not necessarily address memory issues. Switching to Athena (option A) would change the architecture and is not a fix for the existing Glue job.

Practice this question →

428

MCQmedium

A data engineer needs to load data from an on-premises Oracle database to Amazon S3 daily. The table is 500 GB and grows by 50 MB per day. The load must capture only new and changed rows since the last run. Which solution is MOST cost-effective and requires the least maintenance?

A.Write a custom Python script on EC2 to query the Oracle redo logs and upload to S3

B.Export the entire table to CSV daily using a script and upload to S3

C.Use AWS Glue ETL job with a JDBC connection and a timestamp filter

D.Use AWS Database Migration Service (DMS) with ongoing replication (CDC)

AnswerD

DMS supports CDC and can capture only changes, minimizing cost and effort.

Why this answer

AWS DMS with ongoing replication (CDC) is the most cost-effective and low-maintenance solution because it continuously captures only new and changed rows from the Oracle source using its built-in CDC mechanism (reading redo logs), without requiring custom scripting or full table exports. It automatically handles schema changes, resumability, and incremental loading to S3, minimizing operational overhead and data transfer costs.

Exam trap

The trap here is that candidates often choose AWS Glue with a timestamp filter (Option C) because it seems simpler, but they overlook that Glue still performs a full table scan via JDBC to apply the filter, which is inefficient for large tables and does not provide true CDC from redo logs, unlike DMS's native log-based replication.

How to eliminate wrong answers

Option A is wrong because writing a custom Python script on EC2 to parse Oracle redo logs is complex to implement, requires deep Oracle internals knowledge, and demands ongoing maintenance for log format changes and error handling, making it neither cost-effective nor low-maintenance. Option B is wrong because exporting the entire 500 GB table daily to CSV is extremely inefficient, wastes significant compute and network resources, and incurs high S3 storage costs for unchanged data, failing the 'only new and changed rows' requirement. Option C is wrong because while AWS Glue with a timestamp filter can capture incremental changes, it requires the source table to have a reliable, monotonically increasing timestamp column and still performs a full JDBC scan of the table to filter rows, which is inefficient for a 500 GB table and does not natively support change data capture from redo logs.

Practice this question →

429

MCQhard

Refer to the exhibit. A CloudFormation stack outputs the Glue job name and S3 bucket names. The Glue job transforms CSV files from the raw bucket to Parquet in the processed bucket. However, the Glue job is failing with an error that it cannot write to the processed bucket. What is the most likely cause?

A.The Glue job does not have permission to write to the processed bucket

B.The raw data bucket is in a different region

C.The Glue job is not using the correct worker type

D.The Glue job is using an incorrect file format

AnswerA

Missing s3:PutObject on processed-bucket.

Why this answer

The most likely cause is that the Glue job's IAM role lacks the necessary permissions (e.g., s3:PutObject, s3:ListBucket) on the processed bucket. AWS Glue jobs require an IAM role with policies that grant write access to the target S3 bucket; without these permissions, the job fails with a write error. This is a common misconfiguration when the role is scoped only to read from the raw bucket.

Exam trap

The DEA-C01 exam often tests the misconception that S3 write failures are caused by region mismatches or file format issues, but the actual trap is that candidates overlook the IAM permission layer and attribute the error to non-permission factors like worker type or data format.

How to eliminate wrong answers

Option B is wrong because cross-region access to S3 buckets is fully supported and does not cause a write permission error; the error message specifically indicates a write failure, not a connectivity or region mismatch. Option C is wrong because the worker type (e.g., G.1X, G.2X) affects memory and compute capacity, not S3 write permissions; an incorrect worker type would cause performance or OOM issues, not a bucket write error. Option D is wrong because using an incorrect file format (e.g., specifying Parquet when the output is CSV) would cause a format conversion error, not a permission-denied write error; the error message explicitly states it cannot write to the bucket, pointing to an access control issue.

Practice this question →

430

Multi-Selecthard

A company is ingesting data from multiple sources into Amazon S3 using AWS Glue. The data is then transformed using Apache Spark on Amazon EMR. The data engineer wants to reduce the cost of storing and processing data by compressing the ingested files. Which THREE file formats support compression and are commonly used with Spark? (Choose THREE.)

Select 3 answers

A.ORC

B.Parquet

C.JSON

D.CSV

E.Avro

AnswersA, B, E

ORC is a columnar format that supports compression and is optimized for Hive/Spark.

Why this answer

Correct options: A, B, and E. ORC, Parquet, and Avro all support compression and are commonly used with Spark. These formats are columnar (ORC and Parquet) or row-based with efficient compression (Avro), making them suitable for analytics.

JSON and CSV support compression but are not columnar and less efficient for Spark processing; they are not the best choices for reducing storage and processing costs in this context.

Practice this question →

431

MCQhard

The Glue job attempts to read data from 's3://my-data-bucket/input/' and write to 's3://my-data-bucket/output/'. It also tries to update a table in the Glue Data Catalog. The job fails with an access denied error. What is the MOST likely cause?

A.The IAM role is missing the 's3:ListBucket' permission on the bucket.

B.The 'glue:UpdateTable' action is not allowed on the specific table.

C.The policy is missing a condition key for the S3 bucket.

D.The resource ARN does not include the bucket itself; it only covers objects.

AnswerA

Glue needs ListBucket to read the list of objects in the prefix.

Why this answer

The Glue job reads from 's3://my-data-bucket/input/' and writes to 's3://my-data-bucket/output/'. For S3 read/write operations, the IAM role must have 's3:ListBucket' permission on the bucket itself (my-data-bucket) to allow listing of objects, in addition to 's3:GetObject' and 's3:PutObject' on the object ARN. Without 's3:ListBucket', the job cannot enumerate objects in the input prefix, leading to an access denied error.

Exam trap

The trap here is that candidates focus on the Glue Data Catalog update action (Option B) or overly complex condition keys (Option C), but the immediate failure is due to the missing 's3:ListBucket' permission, which is a fundamental S3 permission required for any read operation that involves listing objects in a bucket.

How to eliminate wrong answers

Option B is wrong because the error is an access denied error during S3 operations, not a Data Catalog update failure; if 'glue:UpdateTable' were missing, the error would occur later and be specific to the Glue API. Option C is wrong because missing a condition key would not cause a blanket access denied error unless the condition explicitly denies access; the error here is due to missing permissions, not condition key misconfiguration. Option D is wrong because even if the resource ARN covers only objects (e.g., 'arn:aws:s3:::my-data-bucket/*'), the 's3:ListBucket' permission must be granted on the bucket ARN (e.g., 'arn:aws:s3:::my-data-bucket') to allow listing; missing this causes the access denied error.

Practice this question →

432

MCQmedium

A retail company uses AWS Glue ETL jobs to process sales data from an S3 data lake. The source data is partitioned by year/month/day in CSV format. The Glue job reads the latest day's data, performs transformations (e.g., cleaning, aggregating), and writes the results to a separate S3 bucket. The job runs on a schedule every day at 2 AM. Recently, the job has been failing intermittently with the error 'AnalysisException: Path does not exist: s3://source-bucket/year=2024/month=02/day=30/'. The engineer verifies that the folder 'day=30' does not exist because February has only 28 days in 2024. The job is reading data from a hardcoded path. The company expects the job to handle variable days per month automatically. What should the engineer do to fix the issue?

A.Modify the script to use Spark SQL with manual partition pruning based on current date

B.Add a try-catch block in the script to skip missing partitions

C.Increase the job's retry count and set a timeout

D.Use a Glue crawler to populate the Data Catalog and use dynamic frame from_catalog with partition predicates

AnswerD

The crawler discovers existing partitions, and dynamic frame reads only available partitions.

Why this answer

Using a Glue crawler to populate the Data Catalog and then using dynamic frame with from_catalog allows Glue to automatically discover all existing partitions. This eliminates the need for hardcoded paths and handles variable days per month. Option A (Spark SQL with manual partition pruning) still requires manual handling of partitions.

Option B (try-catch) is a workaround but does not fix the root cause. Option C (increasing retries) does not address the missing partition issue.

Practice this question →

433

MCQhard

A company ingests JSON data from an S3 bucket into a Glue ETL job. The data contains nested structures and arrays. The team wants to flatten the data into a tabular format for analysis in Athena. Which Glue transformation is appropriate?

A.Map

B.Relationalize

C.Filter

D.DropNullFields

AnswerB

Relationalize transforms nested JSON into relational tables suitable for querying.

Why this answer

The Relationalize transformation is specifically designed to flatten nested JSON and arrays into a tabular format suitable for Athena. Option A (Map) applies a function to each record but does not flatten structures. Option C (Filter) selects rows based on a condition.

Option D (DropNullFields) removes null fields but does not address nested structures.

Practice this question →

434

MCQmedium

A company uses AWS Glue to transform data in Amazon S3. The transformation logic is complex and involves multiple steps. The data engineer wants to implement a workflow that handles dependencies and retries on failure. Which AWS service should be used to orchestrate the Glue jobs?

A.AWS Step Functions

B.AWS Lambda

C.Amazon Managed Workflows for Apache Airflow (MWAA)

D.Amazon CloudWatch Events

AnswerA

Correct. AWS Step Functions can orchestrate multiple Glue jobs with error handling and retries.

Why this answer

AWS Step Functions is the best choice for orchestrating Glue jobs with dependencies and retries.

Practice this question →

435

MCQhard

A data engineer is troubleshooting an AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but writes duplicate rows into Redshift. The source data is static and does not contain duplicates. Which configuration change is most likely to resolve this issue?

A.Enable the 'upsert' feature in the Redshift connection by setting 'update' to true.

B.Modify the job to use the 'postactions' option with a SQL statement that deletes duplicates before final insert.

C.Use partition pruning on the S3 source to reduce the number of files read.

D.Increase the number of DPUs (Data Processing Units) allocated to the job.

AnswerB

Using postactions to perform a MERGE or delete duplicates after staging can ensure idempotent writes.

Why this answer

The job runs successfully but writes duplicate rows because AWS Glue's Spark-based ETL jobs can retry tasks on failure, and when writing to Redshift using the JDBC connector, the default behavior is to append data without deduplication. Using the 'postactions' option with a SQL DELETE statement that removes duplicates before the final INSERT ensures that only unique rows remain, resolving the duplication without altering the source data.

Exam trap

The trap here is that candidates often assume duplicate rows come from the source data or a misconfiguration in the write mode, but the real cause is the default append behavior combined with Spark task retries, and the solution is to use post-write deduplication rather than changing the write mode or source processing.

How to eliminate wrong answers

Option A is wrong because enabling 'upsert' with 'update' to true is used for merging data based on a key, but it does not prevent duplicate rows from being inserted; it only updates existing rows if a key matches, and the source data has no duplicates, so this would not fix the issue. Option C is wrong because partition pruning on the S3 source reduces the number of files read but does not address the duplication caused by job retries or write behavior; it optimizes performance, not data integrity. Option D is wrong because increasing the number of DPUs allocates more compute resources to the job, which can improve performance but does not prevent duplicate writes; duplication is a logic or configuration issue, not a resource constraint.

Practice this question →

436

MCQmedium

A company ingests streaming data from IoT devices into Amazon Kinesis Data Streams. The data must be transformed in real-time using custom Python code before being stored in Amazon S3. Which AWS service should be used to perform this transformation?

A.Amazon EMR

B.Amazon Kinesis Data Firehose

C.AWS Glue

D.Amazon Kinesis Data Analytics for Apache Flink

AnswerD

Kinesis Data Analytics for Apache Flink allows running Flink applications that can process streaming data with custom Python code.

Why this answer

Amazon Kinesis Data Analytics for Apache Flink enables real-time stream processing with custom Python code via Apache Flink's Python API, making it suitable for complex transformations. Option A (Amazon EMR) is wrong as it requires significant setup and is not a fully managed streaming service. Option B (Amazon Kinesis Data Firehose) is wrong because although it can invoke Lambda for simple transformations, it is limited in complexity and not designed for rich Python custom logic.

Option C (AWS Glue) is wrong because it is primarily a batch ETL service and lacks native real-time stream processing capabilities.

Practice this question →

437

MCQhard

A company runs a daily batch ETL job using AWS Glue. The job processes 500 GB of data from Amazon RDS to Amazon S3. The job currently uses a single DPU and takes 6 hours to complete. The team wants to reduce runtime to under 1 hour without increasing costs significantly. Which approach should they use?

A.Change the job type from Python to Spark.

B.Use multiple Glue jobs triggered sequentially.

C.Increase the RDS instance size to improve read throughput.

D.Use AWS Glue Spark job with 100 workers.

AnswerD

More workers enable parallelism, reducing runtime.

Why this answer

AWS Glue Spark jobs can parallelize data processing across multiple workers, dramatically reducing runtime. With 100 workers, the job can process the 500 GB dataset in parallel, achieving sub-1-hour runtime while keeping costs relatively low since Glue charges per DPU-second and the total DPU-seconds may be similar to the original 6-hour single-DPU job.

Exam trap

The trap here is that candidates might think increasing parallelism (Option D) is too expensive, but Glue's pay-per-DPU-second model means a job with 100 workers running for 1 hour costs roughly the same as 1 worker running for 100 hours, so the total cost is similar, not significantly higher.

How to eliminate wrong answers

Option A is wrong because changing from Python to Spark alone does not add parallelism; the job still runs on a single DPU unless the number of workers is increased. Option B is wrong because running multiple Glue jobs sequentially would increase total runtime, not reduce it, as each job would still process data serially. Option C is wrong because the bottleneck is Glue's processing capacity, not RDS read throughput; increasing RDS instance size would not significantly reduce Glue job runtime since the job already reads 500 GB over 6 hours, and the read rate is not the limiting factor.

Practice this question →

438

Multi-Selecthard

A company uses AWS DMS to replicate data from an Amazon RDS for MySQL database to Amazon S3. Which TWO configurations are required to enable continuous change data capture (CDC) from MySQL?

Select 2 answers

A.Ensure the S3 bucket is in the same AWS Region as the source database

B.Grant REPLICATION CLIENT and REPLICATION SLAVE privileges to the DMS user

C.Enable binary logging (binlog) on the MySQL source database

D.Enable versioning on the target S3 bucket

E.Configure the MySQL source to be Multi-AZ

AnswersB, C

Required for DMS to read binary logs.

Why this answer

Correct options: B and C. For AWS DMS to perform continuous change data capture (CDC) from a MySQL source, binary logging (binlog) must be enabled on the source database (option C) to capture changes. Additionally, the MySQL user used by DMS must be granted the REPLICATION CLIENT and REPLICATION SLAVE privileges (option B) to read the binlog and stream changes.

Option D (S3 bucket versioning) is not required for DMS CDC. Option A (same Region) is not a requirement. Option E (Multi-AZ) is not needed for CDC.

Therefore, B and C are the required configurations.

Practice this question →

439

MCQeasy

A company needs to transform JSON data from an S3 bucket into a structured format for Amazon Redshift. The transformation should be done serverlessly. Which service should be used?

A.AWS Glue

B.Amazon EMR

C.Amazon Athena

D.AWS Lambda

AnswerA

Glue provides serverless ETL capabilities.

Why this answer

AWS Glue is the correct choice because it is a fully managed, serverless ETL service designed specifically for transforming and preparing data for analytics, including converting JSON to structured formats like Parquet or ORC for Amazon Redshift. It can crawl the S3 source, infer schemas, and run Spark-based transformation jobs without provisioning any infrastructure, aligning perfectly with the serverless requirement.

Exam trap

The trap here is that candidates often confuse Amazon Athena's serverless SQL querying capability with ETL transformation, but Athena cannot transform or write data into a different format for Redshift—it only reads and queries data in place.

How to eliminate wrong answers

Option B (Amazon EMR) is wrong because it requires provisioning and managing EC2 clusters, which is not serverless; it is a managed Hadoop framework but still involves underlying infrastructure. Option C (Amazon Athena) is wrong because it is a serverless query engine for analyzing data directly in S3 using SQL, not a transformation service for converting JSON to a structured format for Redshift. Option D (AWS Lambda) is wrong because it is designed for short-running, event-driven functions (max 15-minute execution time) and is not suitable for large-scale ETL transformations on big datasets, which typically require longer-running jobs.

Practice this question →

440

MCQeasy

A company uses Amazon S3 Event Notifications to trigger a Lambda function that processes incoming files. Recently, the Lambda function has been timing out for large files (>100 MB). The data engineer wants to improve the pipeline to handle large files reliably. Which solution is the MOST scalable and cost-effective?

A.Use S3 Event Notification to send to an SQS queue, then have Lambda poll the queue

B.Use Amazon SNS to fan out the event to multiple Lambda functions

C.Use AWS Step Functions to orchestrate multiple Lambda functions for parallel processing

D.Increase the Lambda timeout to 15 minutes

AnswerA

SQS decouples and buffers events, allowing Lambda to process at a manageable rate.

Why this answer

Decoupling S3 event notifications via an SQS queue allows Lambda to poll messages at its own pace, preventing timeouts from large files. The SQS queue acts as a buffer, enabling Lambda to process files asynchronously and scale based on the queue depth without being constrained by the synchronous S3 trigger timeout (typically 15 minutes for Lambda, but large files can still cause issues with concurrent execution limits). This approach is both scalable and cost-effective, as it avoids idle wait time and allows Lambda to process files in smaller chunks or with longer execution times as needed.

Exam trap

The trap here is that candidates assume increasing Lambda timeout is the simplest fix, but the DEA-C01 exam tests understanding of decoupling patterns (SQS) to handle variable workloads and avoid synchronous invocation bottlenecks.

How to eliminate wrong answers

Option B is wrong because fanning out via SNS to multiple Lambda functions does not address the root cause of timeouts; it merely duplicates the same synchronous invocation pattern, potentially overwhelming Lambda concurrency limits and increasing costs without improving reliability for large files. Option C is wrong because AWS Step Functions orchestrate multiple Lambda functions for parallel processing, which adds complexity and cost (per state transition) without solving the timeout issue for a single large file; Step Functions are better for workflows with multiple steps, not for buffering or retry logic. Option D is wrong because simply increasing the Lambda timeout to 15 minutes does not address scalability or cost; it risks exhausting Lambda concurrency limits (e.g., 1,000 concurrent executions by default) and incurs higher costs for idle time, while still failing if the file processing exceeds 15 minutes or if multiple large files arrive simultaneously.

Practice this question →

441

MCQmedium

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data from a website. The data is consumed by an Amazon Kinesis Data Analytics for Apache Flink application that performs real-time analytics. The Flink application writes its results to an Amazon S3 bucket. The company has noticed that the Flink application is experiencing high checkpoint failure rates, causing delays. The CloudWatch metrics show that the checkpoint size is large and increasing. The data engineer needs to reduce the checkpoint size. Which action should the data engineer take?

A.Decrease the checkpoint interval to reduce the amount of state accumulated.

B.Reduce the parallelism of the Flink application.

C.Increase the state time-to-live (TTL) configuration to retain state longer.

D.Enable incremental checkpointing in the Flink application to only write changes since the last checkpoint.

AnswerD

Incremental checkpoints reduce size and improve performance.

Why this answer

Enabling incremental checkpointing in Flink reduces the amount of data written per checkpoint by only writing changes since the last checkpoint. Option A is wrong because reducing parallelism may increase load per operator. Option B is wrong because decreasing checkpoint interval increases frequency, not size.

Option C is wrong because state TTL does not directly reduce checkpoint size.

Practice this question →

442

MCQmedium

A company is using Amazon Kinesis Data Firehose to ingest log data from web servers into an Amazon S3 bucket. The data is then queried by Amazon Athena. The company has noticed that the Athena queries are slow and expensive. The data engineer wants to optimize the storage format to improve query performance and reduce costs. Which configuration change should the data engineer make to the Firehose delivery stream?

A.Increase the buffer interval to 600 seconds and buffer size to 128 MB to create larger files.

B.Change the output format to ORC and enable GZIP compression.

C.Enable S3 server access logs to track query patterns.

D.Enable data transformation in Firehose to convert JSON to Parquet format with Snappy compression.

AnswerD

Parquet is columnar and efficient for Athena.

Why this answer

Enable data transformation in Firehose to convert JSON to Parquet format with Snappy compression. Parquet is a columnar storage format that significantly improves query performance in Athena by reducing the amount of data scanned per query. Snappy compression provides efficient compression and decompression, reducing storage costs and improving I/O.

Option A is incorrect because increasing buffer interval and size simply creates larger files but does not change the storage format; the data remains in its original format (likely JSON or CSV), which is less efficient for columnar querying. Option B is incorrect because while ORC is also a columnar format, Parquet is more commonly used with Athena and offers better integration; additionally, GZIP compression is not as efficient as Snappy for Parquet files. Option C is incorrect because enabling S3 server access logs would track requests to the S3 bucket but does not optimize the data format or improve query performance; it adds additional cost and storage overhead.

Practice this question →

443

MCQeasy

A company wants to ingest data from thousands of IoT devices into AWS for real-time analytics. The data is in JSON format and each device sends about 1 KB every second. Which service should be used as the primary ingestion point?

A.AWS IoT Core

B.Amazon Kinesis Data Firehose

C.Amazon SQS

D.Amazon Kinesis Data Streams

AnswerD

Handles high-volume streaming data.

Why this answer

Amazon Kinesis Data Streams (D) is the correct choice because it is designed for real-time streaming of large amounts of data from many producers, such as thousands of IoT devices. Each device sends 1 KB per second, resulting in ~1 MB/s total throughput, which Kinesis Data Streams can handle with sharding. It supports multiple consumers for real-time analytics.

AWS IoT Core (A) is for device management and MQTT messaging, not a general-purpose ingestion point for analytics. Kinesis Data Firehose (B) is for loading streaming data into storage, but it does not support multiple real-time consumers and has a minimum 60-second buffer latency. Amazon SQS (C) is a message queue for decoupled applications, not built for high-throughput streaming analytics.

Exam trap

Confusing the purpose of Kinesis Data Streams (real-time ingestion with multiple consumers) vs. Kinesis Data Firehose (delivery to storage with latency) is common. Also, remember that AWS IoT Core is a device gateway, not a data ingestion service for analytics.

Practice this question →

444

MCQeasy

A company uses AWS Glue ETL jobs to transform data and load it into Amazon Redshift. The jobs are failing with 'Out of Memory' errors. What is the most cost-effective way to resolve this issue without changing the transformation logic?

A.Increase the number of G.1X workers in the Glue job configuration.

B.Use Amazon Redshift Spectrum to query data directly from S3 without transformation.

C.Change the worker type to G.2X and keep the same number of workers.

D.Switch the job from Python to Scala.

AnswerA

More workers increase parallelism and total memory.

Why this answer

Increasing the number of G.1X workers (DPUs) adds parallelism, allowing the job to handle more data in memory without changing logic, and is cost-effective since G.1X workers are cheaper than G.2X. Option B is wrong: Redshift Spectrum is for querying data directly from S3, not for fixing memory issues in Glue ETL jobs. Option C is wrong: Changing to G.2X workers increases memory per worker but is more expensive than adding more G.1X workers; the goal is cost-effective.

Option D is wrong: Switching to Scala does not directly address memory issues and may require code changes.

Practice this question →

445

MCQmedium

A data engineer is ingesting streaming data from an IoT fleet into Amazon Kinesis Data Streams. The data must be transformed in real-time and loaded into an Amazon Redshift cluster. Which solution minimizes operational overhead?

A.Use Kinesis Data Firehose with a Lambda transformation function

B.Use AWS Glue ETL jobs running continuously

C.Use Kinesis Client Library (KCL) to consume and transform data, then write to Redshift using COPY

D.Use AWS Direct Connect to stream data directly into Redshift

AnswerA

Firehose handles buffering, transformation via Lambda, and direct delivery to Redshift.

Why this answer

Kinesis Data Firehose is the fully managed service for loading streaming data into Redshift with near-real-time latency. By attaching a Lambda transformation function, you can perform lightweight data transformations (e.g., JSON flattening, field masking) without managing any compute infrastructure. This combination eliminates the need to provision or tune any servers, clusters, or consumer applications, minimizing operational overhead.

Exam trap

The trap here is that candidates often confuse Kinesis Data Streams with Kinesis Data Firehose, assuming they must write a custom consumer (KCL) to transform data, when Firehose with Lambda provides a fully managed, serverless alternative that reduces operational overhead.

How to eliminate wrong answers

Option B is wrong because AWS Glue ETL jobs are designed for batch-oriented, schema-on-read transformations and are not optimized for continuous, low-latency streaming ingestion into Redshift; running them continuously would incur high cost and operational complexity. Option C is wrong because using the Kinesis Client Library (KCL) requires you to deploy, scale, and manage your own consumer application (e.g., on EC2 or ECS) to consume the stream, transform data, and issue COPY commands, which adds significant operational overhead compared to a serverless Firehose. Option D is wrong because Direct Connect is a dedicated network connection between on-premises and AWS, not a data ingestion service; it cannot stream data directly into Redshift and provides no transformation capability.

Practice this question →

446

MCQhard

A financial services company is ingesting trade data from multiple exchanges via Amazon Kinesis Data Streams. Each shard receives data from multiple exchanges, and a consumer application (using KCL) processes the data. The company needs to ensure that trades from the same exchange are processed in order. However, the current implementation distributes records to shards using a random partition key, causing trades from the same exchange to be spread across shards and processed out of order. The team must enforce ordering per exchange without significantly reducing throughput. What should the team do?

A.Implement a custom sequence number in the application to reorder after processing.

B.Use a single shard for all data to guarantee order.

C.Use the exchange ID as the partition key when putting records into the stream.

D.Increase the number of shards to 10 per exchange.

AnswerC

Ensures same exchange goes to same shard, preserving order.

Why this answer

Using the exchange ID as the partition key ensures all trades from the same exchange go to the same shard, preserving order. Option A is wrong because increasing shard count would further spread data and break ordering. Option B is wrong because using a single shard would preserve order but reduce throughput due to shard limits.

Option D is wrong because implementing a custom sequencer is complex and unnecessary.

Practice this question →

447

MCQhard

A company uses Amazon Kinesis Data Analytics (now Managed Service for Apache Flink) to run a Flink application on streaming data. The application fails with 'OutOfMemoryError: Java heap space'. The data volume is 10 MB/s. What is the most likely cause and solution?

A.The data contains records larger than 1 MB; split records into smaller chunks.

B.Checkpointing is enabled too frequently; reduce checkpoint interval.

C.The Flink application is not suitable for 10 MB/s throughput; use Kinesis Data Firehose instead.

D.The application's Parallelism is too low; increase the number of Parallelism and KPUs.

AnswerD

Low parallelism causes data to accumulate in operator buffers, leading to OOM.

Why this answer

The OutOfMemoryError in a Flink application on Amazon Kinesis Data Analytics (Managed Service for Apache Flink) is most likely due to insufficient parallelism to handle the 10 MB/s data volume. Increasing parallelism distributes the workload across more KPUs (Kinesis Processing Units), reducing memory pressure per operator and preventing heap exhaustion. Option D directly addresses this by scaling resources to match throughput.

Exam trap

The trap here is that candidates often misdiagnose an OOM as a record size issue (Option A) or a checkpointing problem (Option B), when in fact the root cause is insufficient parallelism to handle the sustained throughput, which is a common scaling pitfall in Flink on Kinesis Data Analytics.

How to eliminate wrong answers

Option A is wrong because Kinesis Data Analytics for Apache Flink supports records up to 1 MB, and the error is heap space, not record size; splitting records would not resolve memory exhaustion from high throughput. Option B is wrong because frequent checkpointing can increase memory usage due to state snapshots, but reducing the interval would exacerbate the problem, not fix it; the core issue is insufficient parallelism. Option C is wrong because Flink is well-suited for 10 MB/s throughput; Kinesis Data Firehose is a serverless ingestion service that cannot run custom Flink applications, so it is not a replacement for a Flink streaming job.

Practice this question →

448

Multi-Selectmedium

A company is using AWS Glue to run ETL jobs that transform data from S3 to Redshift. The jobs are failing intermittently with out-of-memory errors. Which THREE actions can help resolve this issue? (Choose THREE.)

Select 3 answers

A.Increase the number of DPUs allocated to the Glue job

B.Use S3 Select to filter data before reading into the Glue job

C.Use Spark's 'coalesce' function to reduce the number of partitions

D.Optimize the transformation logic to use less memory, for example by filtering early

E.Use a larger worker type, such as G.2X

AnswersA, D, E

More DPUs provide more memory and compute resources.

Why this answer

Increasing the number of DPUs allocated to the Glue job provides more memory and compute resources for the Spark executors, directly addressing out-of-memory errors by allowing larger datasets to be processed without exceeding heap limits. This is a standard scaling approach for memory-intensive ETL workloads in AWS Glue.

Exam trap

The trap here is that candidates often confuse reducing data volume (S3 Select) with increasing memory capacity, or mistakenly believe coalescing partitions always reduces memory usage, when in fact it can concentrate data and exacerbate OOM errors.

Practice this question →

449

MCQhard

A company has a 100 TB dataset stored on-premises in a Hadoop cluster. They want to ingest this data into Amazon S3 for processing with AWS Glue. The company has a limited time window and a slow internet connection. Which strategy is MOST appropriate?

A.Use AWS Snowball Edge to physically ship the data to AWS.

B.Use AWS DataSync over the existing internet connection.

C.Use Amazon S3 Transfer Acceleration to speed up the upload.

D.Use AWS Direct Connect to establish a high-bandwidth connection.

AnswerA

Snowball Edge can handle 100 TB offline, bypassing network limitations.

Why this answer

AWS Snowball Edge is the most appropriate strategy because the dataset is 100 TB, the time window is limited, and the internet connection is slow. Snowball Edge provides a physical storage device that can be shipped to AWS, bypassing network bandwidth constraints entirely. This approach is designed for large-scale data transfers (typically over 10 TB) where network transfer would be impractical or exceed the available time window.

Exam trap

The trap here is that candidates may overestimate the effectiveness of network acceleration techniques (like Transfer Acceleration or Direct Connect) for extremely large datasets, failing to recognize that physical shipping is the only viable option when bandwidth and time are severely constrained.

How to eliminate wrong answers

Option B is wrong because AWS DataSync relies on the existing internet connection, which is slow and would take an excessively long time to transfer 100 TB, likely exceeding the limited time window. Option C is wrong because Amazon S3 Transfer Acceleration uses edge locations and optimized network paths, but it still depends on the underlying internet connection speed; a slow connection will remain a bottleneck, and it is not designed for petabyte-scale offline transfers. Option D is wrong because AWS Direct Connect requires establishing a dedicated network connection, which involves significant lead time for setup and does not solve the immediate problem of a slow internet connection; it also still transfers data over a network, which for 100 TB would be time-consuming even at high bandwidth.

Practice this question →

450

MCQmedium

A company uses AWS Glue to process data from multiple sources. The data is stored in an Amazon S3 data lake. The company needs to transform the data using a custom Python library that is not available in the default Glue environment. What is the MOST efficient way to make this library available to the Glue jobs?

A.Manually install the library on each node in the Glue cluster by editing the bootstrap script.

B.Upload the library as a .whl file to Amazon S3 and reference it in the Glue job's --additional-python-modules parameter.

C.Create a custom Docker image with the library and use it in AWS Glue for Ray.

D.Use a shell command in the Glue job script to run 'pip install <library>' before the job runs.

AnswerB

This is the recommended way to add custom libraries to Glue jobs.

Why this answer

AWS Glue supports adding custom Python libraries by uploading a .whl file to Amazon S3 and referencing it via the `--additional-python-modules` job parameter. This method is the most efficient as it requires no manual node configuration, no custom Docker images, and no runtime pip installs, ensuring the library is automatically distributed to all worker nodes before the job executes.

Exam trap

The trap here is that candidates may think running 'pip install' directly in the script (Option D) is acceptable, but AWS explicitly recommends using the `--additional-python-modules` parameter for efficiency and reliability, as runtime pip installs can fail due to network timeouts or missing build dependencies.

How to eliminate wrong answers

Option A is wrong because manually editing bootstrap scripts to install the library on each node is inefficient, error-prone, and not scalable; Glue manages cluster lifecycle automatically, so manual node-level modifications are not recommended and can be lost on auto-scaling events. Option C is wrong because AWS Glue for Ray is a specific runtime for distributed Python and Ray-based workloads, not a general-purpose Glue ETL job; using a custom Docker image for Ray adds unnecessary complexity and is not the standard approach for standard Glue ETL jobs. Option D is wrong because running 'pip install' inside the Glue job script is inefficient, adds runtime overhead, may fail due to network restrictions or permissions, and is not the intended way to manage dependencies in Glue; the library must be pre-packaged and referenced via the job parameters.

Practice this question →

← PreviousPage 6 of 8 · 591 questions totalNext →

Ready to test yourself?

Try a timed practice session using only Data Ingestion and Transformation questions.

Start 20-question session