DEA-C01 Data Ingestion and Transformation Practice Test 15 — 15 Questions

Question 1

A data engineer is troubleshooting an AWS Glue ETL job that fails with an OutOfMemory error when processing large JSON files from Amazon S3. The files contain deeply nested structures. Which approach should the engineer take to resolve this issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the DPU (Data Processing Unit) per worker. Increasing the DPU per worker allocates more memory per worker, directly addressing the OutOfMemory error when processing large files. Option A is incorrect because the `recurse` option is not a valid argument for `getResolvedOptions`, which is used for retrieving job parameters. Option B is incorrect because increasing the number of workers adds parallelism but does not increase memory per worker; OOM occurs per worker. Option D is incorrect because decreasing partitions reduces parallelism, potentially causing each partition to be larger, worsening the memory issue.

Answer

Use the `recurse` option with `getResolvedOptions` to limit recursion

Answer

Increase the number of workers in the Glue job configuration

Answer

Decrease the number of partitions while reading the data

Question 2

A company is using AWS Lake Formation to manage permissions on data in Amazon S3. They need to ingest data from an external source into a new database 'sales_db' and a table 'transactions' using AWS Glue. The IAM role used by Glue must have the minimal permissions to create the database and table in the Data Catalog and write data to the S3 location. Which combination of permissions should be granted?

Accepted Answer

Lake Formation permissions: `CREATE_DATABASE` on the catalog, `CREATE_TABLE` on `sales_db`, and S3 location permission. When Lake Formation manages permissions, the Glue job's IAM role must be granted Lake Formation permissions (CREATE_DATABASE on the catalog, CREATE_TABLE on sales_db) and a data location permission for the S3 path. These Lake Formation grants allow the role to create the database and table in the Data Catalog and write to the registered S3 location, which is the minimal set required.

Answer

IAM policy with `glue:CreateDatabase`, `glue:CreateTable`, and `s3:PutObject`

Answer

IAM policy with `lakeformation:GrantPermissions` on the database and table

Answer

IAM policy with `s3:GetObject` and `s3:PutObject` on the target location

Question 3

A company needs to ingest data from an on-premises SQL Server database into Amazon Redshift. The data volume is less than 1 TB and the network bandwidth is limited. Which AWS service should be used for the initial full load?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is designed for migrating databases to AWS, including from on-premises SQL Server to Amazon Redshift. It supports ongoing replication and can handle the initial full load efficiently over limited bandwidth by compressing data and using change data capture (CDC) for incremental updates. For datasets under 1 TB, DMS is cost-effective and simpler than physical shipping options.

Answer

AWS Snowball Edge

Answer

Amazon S3 Transfer Acceleration

Answer

AWS Direct Connect

Question 4

A data engineer is designing a data ingestion pipeline for real-time clickstream data using Amazon Kinesis Data Streams. The data must be transformed using AWS Lambda and then stored in Amazon S3 in Parquet format. Which Kinesis client library configuration should be used to minimize the number of Lambda invocations while ensuring data is processed within 60 seconds?

Accepted Answer

Set batch size to 10000 records and batch window to 60 seconds. A batch size of 10,000 records combined with a batch window of 60 seconds maximizes the number of records per Lambda invocation while respecting the 60-second processing requirement, thereby minimizing the total number of invocations. Option A (batch size 100, no batch window) leads to many small invocations. Option C (batch window 0 seconds) triggers immediate processing, increasing invocation frequency. Option D (batch window 5 seconds) forces more frequent invocations than necessary, even with a large batch size.

Answer

Set batch size to 100 records and disable batch window

Answer

Set batch size to 100 records and batch window to 0 seconds

Answer

Set batch size to 10000 records and batch window to 5 seconds

Question 5

A company is using Amazon MSK (Managed Streaming for Apache Kafka) to ingest real-time data. They need to transform the data using custom Java code before writing to Amazon S3. The transformation must be fault-tolerant and exactly-once semantics are required. Which AWS service should be used?

Accepted Answer

Kafka Connect with S3 Sink Connector. Kafka Connect with S3 Sink Connector supports exactly-once semantics and custom transformations via Single Message Transforms (SMTs) or custom connectors. Option A (Amazon EMR with Spark Streaming) is for batch processing, not real-time exactly-once from MSK. Option B (AWS Lambda) does not provide exactly-once semantics from MSK. Option D (Kinesis Data Analytics for Apache Flink) is for Flink, not Kafka.

Answer

Amazon EMR with Spark Streaming

Answer

AWS Lambda consumer for MSK

Answer

Kinesis Data Analytics for Apache Flink

Question 6

A company needs to ingest data from Amazon S3 into Amazon Redshift for analytics. The data arrives in CSV format with headers and may contain duplicate rows. Which Redshift command should be used to load the data while handling duplicates?

Accepted Answer

Use a staging table with COPY and then MERGE into the target table. The correct approach to handle duplicates when loading data from S3 into Redshift is to use a staging table with the COPY command, followed by a MERGE (or UPSERT) operation to insert distinct rows into the target table. Option A is incorrect because `REMOVEDUPLICATES` is not a valid COPY option. Option B is incorrect because INSERT INTO ... SELECT DISTINCT via Spectrum does not use Redshift's native ingestion and may be slower. Option D is incorrect because CREATE TABLE AS SELECT DISTINCT does not integrate with existing tables and is not a standard loading pattern.

Answer

COPY command with the `REMOVEDUPLICATES` option

Answer

INSERT INTO ... SELECT DISTINCT from S3 via Spectrum

Answer

CREATE TABLE AS SELECT DISTINCT from the S3 bucket

Question 7

A data engineer is designing a data pipeline that uses AWS Glue to transform data stored in Amazon S3. The transformation logic must be written in Python and should handle schema evolution automatically. Which THREE features or configurations should the engineer use? (Select THREE.)

Accepted Answer

Use `applyMapping` transformations. Correct options: B, D, E. AWS Glue DynamicFrames (E) handle schema evolution automatically by allowing schema on read and accommodating changes in data structure. Schema detection in the Glue job (D) enables the job to infer the schema from the data, which is essential for handling evolving schemas. Using `applyMapping` (B) provides explicit control over schema transformations and can be combined with DynamicFrames to manage schema changes. Option A (scheduling a Glue crawler) is meant for updating the Data Catalog, not for within-job schema evolution. Option C (Spark SQL) does not inherently handle schema evolution; it relies on static schemas.

Answer

Schedule a Glue crawler to update the schema

Answer

Use Spark SQL for transformations

Question 8

A company uses AWS Glue to transform data in Amazon S3. The transformation logic is written in Python and references several libraries that are not included in the default Glue environment. Which approach should the data engineer use to make these libraries available?

Accepted Answer

Upload the libraries as a .zip file to an S3 bucket and reference them in the Glue job's Python library path.. AWS Glue allows you to provide custom Python libraries by uploading them as a .zip file to an S3 bucket and referencing the S3 path in the 'Python library path' field of the job. This method is supported for both Glue ETL and Python shell jobs, making custom libraries available at runtime. Option A is incorrect because Glue does not execute arbitrary shell commands like 'pip install' during job initialization; the job script runs in a controlled environment. Option B is incorrect because AWS Lambda layers are specific to Lambda functions and cannot be attached to Glue jobs. Option D is incorrect because the --additional-python-modules parameter, while valid in Glue 3.0 and later, only supports installing packages from PyPI, not custom libraries not published on PyPI.

Answer

Include a requirements.txt file in the Glue job script and run pip install during job initialization.

Answer

Package the libraries in an AWS Lambda layer and attach it to the Glue job.

Answer

Use the --additional-python-modules parameter in the Glue job.

Question 9

A data engineer needs to ingest data from an Amazon RDS for MySQL database into Amazon S3 on a daily basis. The data volume is about 50 GB per day. The engineer wants to minimize the impact on the source database. Which AWS service should be used?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is designed for continuous or scheduled data replication from source databases to targets like S3 with minimal impact on the source. DMS uses change data capture (CDC) or full-load-plus-CDC modes that read from the source's redo/transaction logs rather than issuing heavy SELECT queries, which minimizes load on the RDS MySQL instance. For 50 GB/day, DMS with a replication instance sized appropriately handles the volume efficiently.

Answer

AWS Glue with a JDBC connection

Answer

Amazon Athena Federated Query

Answer

AWS DataSync

Question 10

A data engineer is ingesting data from a third-party API into Amazon S3 using AWS Lambda. The API returns a JSON payload of up to 10 MB per request. The Lambda function runs every minute. Occasionally, the function times out after 15 seconds. What is the most likely cause?

Accepted Answer

The Lambda function's timeout setting is too short.. The Lambda function times out after 15 seconds, which strongly suggests the configured timeout is too short for the API call and processing. The most likely cause is that the timeout setting is lower than the actual time needed, so increasing it (up to 15 minutes) resolves the issue.

Answer

The Lambda function is in a VPC without a NAT gateway, causing network timeouts.

Answer

The Lambda function's memory is too low, causing slow processing.

Answer

The API response size exceeds the Lambda invocation payload limit.

Question 11

Which TWO AWS services can be used to ingest streaming data into Amazon S3 with minimal code? (Choose two.)

Accepted Answer

Amazon Kinesis Data Firehose. Amazon Kinesis Data Firehose (B) is correct because it is a fully managed service that can deliver streaming data directly into Amazon S3 with essentially no code — you simply configure a delivery stream with S3 as the destination, and Firehose handles buffering, compression, encryption, and retries. Amazon MSK with an S3 sink connector (C) is also correct because the managed Kafka Connect S3 sink connector ingests streaming records from Kafka topics into S3 with only configuration rather than custom application code. AWS Lambda (A) is not the best fit here because it requires you to write and maintain function code to read from a stream and write objects to S3. AWS DMS (D) is designed for migrating and replicating databases, not for general streaming ingestion into S3. AWS DataSync (E) is a data-transfer service for moving files between on-premises storage and AWS, not for ingesting streaming data.

Answer

AWS Lambda

Answer

AWS Database Migration Service (DMS)

Answer

AWS DataSync

Question 12

Refer to the exhibit. An IAM policy is attached to a role used by an AWS Glue job. The job fails with an 'AccessDenied' error when trying to write to 's3://my-bucket/output/'. What is the most likely cause?

Accepted Answer

The resource ARN for S3 should include the bucket itself.. The most likely cause is that the S3 resource ARN in the IAM policy is malformed — typically written as 'arn:aws:s3:::my-bucket/*' without also granting the bucket-level ARN 'arn:aws:s3:::my-bucket'. AWS Glue (and the S3 SDK) performs a HeadBucket/ListBucket-style bucket-level check before writing objects, so a policy that only lists object ARNs will produce AccessDenied on the write path. The resource ARN must include both the bucket itself and the bucket's objects.

Answer

The glue:StartJobRun action is not allowed.

Answer

The policy does not grant s3:ListBucket permission.

Answer

The s3:GetObject action is missing.

Question 13

Refer to the exhibit. A data engineer runs this AWS CLI command to create a Glue job. The job processes JSON files in an S3 bucket and writes Parquet files to another bucket. After the first successful run, the job re-processes all input files instead of only new files. What is the most likely cause?

Accepted Answer

The job script does not implement job bookmark support.. The command sets '--job-bookmark-enable' but if the job script does not use the bookmark APIs or implement bookmark support, Glue will not track processed files and will reprocess all input on each run. Option A is incorrect because the ScriptLocation is valid and does not affect bookmark behavior. Option B is incorrect because max-retries does not control reprocessing. Option D is incorrect because the IAM role is specified and permissions for bookmark state are not explicitly shown, but the lack of bookmark support in the script is the issue.

Answer

The ScriptLocation points to an incorrect S3 path.

Answer

The --max-retries parameter is set to 0.

Answer

The IAM role lacks permissions to read bookmark state.

Question 14

A data engineer needs to transform JSON data from an S3 bucket into Parquet format for efficient querying with Amazon Athena. The transformation must be serverless and event-driven. Which approach meets these requirements?

Accepted Answer

Use S3 event notifications to invoke a Lambda function that runs PySpark to convert the data.. S3 event notifications can trigger a Lambda function that launches an AWS Glue ETL job. Glue ETL jobs run PySpark code natively to convert JSON to Parquet, satisfying the serverless and event-driven requirements. Athena only queries existing data, Glue crawlers do not perform transformations, and EMR clusters are not serverless.

Answer

Use AWS Glue with a scheduled crawler to convert the data.

Answer

Use Amazon Athena to convert JSON to Parquet on the fly.

Answer

Use Amazon EMR with a long-running cluster to process S3 data.

Question 15

A company uses Amazon Kinesis Data Firehose to deliver data to an S3 bucket. The data must be delivered within 60 seconds of ingestion. Currently, the delivery takes 3 minutes due to large buffer sizes. How should the engineer adjust the Firehose configuration?

Accepted Answer

Decrease the buffer interval to 60 seconds.. Amazon Kinesis Data Firehose delivers data to S3 based on either a buffer size threshold or a buffer interval (in seconds), whichever is reached first. To ensure delivery within 60 seconds, you must decrease the buffer interval to 60 seconds, which forces Firehose to flush data to S3 every 60 seconds regardless of buffer size. The current 3-minute delay is caused by the buffer interval being larger than 60 seconds, so reducing it directly meets the requirement.

Answer

Increase the buffer interval to 120 seconds.

Answer

Increase the buffer size to 128 MB.

Answer

Decrease the buffer size to 1 MB.