DEA-C01 Data Ingestion and Transformation Practice Test 2 — 15 Questions

Question 1

A financial services company ingests real-time stock trade data from multiple exchanges into Amazon Kinesis Data Streams. Each trade record is a JSON object containing fields: trade_id, symbol, price, quantity, and timestamp. The data is consumed by an AWS Lambda function that performs data validation and enrichment, then writes the processed records to an Amazon DynamoDB table for low-latency querying. Recently, the Lambda function has been timing out and failing to process all records. The Lambda function is configured with a 5-second timeout and 128 MB memory. The average record size is 2 KB, and the stream receives about 1000 records per second. The Lambda function's concurrency limit is 1000. Which set of actions should the data engineer take to resolve the issue without losing data?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records and enable parallelization factor of 10.. Increasing the Lambda timeout and memory addresses the processing bottleneck, while setting the batch size to 100 and enabling a parallelization factor of 10 allows each shard to process up to 10 concurrent batches, dramatically increasing throughput to handle 1000 records/sec (each shard can process 10 batches of 100 records concurrently, yielding 1000 records/sec per shard if the stream has at least 1 shard). This combination ensures no data loss by keeping up with the ingestion rate without exceeding the Lambda concurrency limit of 1000.

Answer

Increase the number of shards in the Kinesis data stream to 20 and keep the Lambda configuration unchanged.

Answer

Replace the Lambda function with a Kinesis Data Firehose delivery stream that writes directly to DynamoDB using a Lambda transformation.

Answer

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records.

Question 2

A data engineer is troubleshooting an AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job runs successfully but writes duplicate rows into Redshift. The source data is static and does not contain duplicates. Which configuration change is most likely to resolve this issue?

Accepted Answer

Modify the job to use the 'postactions' option with a SQL statement that deletes duplicates before final insert.. The job runs successfully but writes duplicate rows because AWS Glue's Spark-based ETL jobs can retry tasks on failure, and when writing to Redshift using the JDBC connector, the default behavior is to append data without deduplication. Using the 'postactions' option with a SQL DELETE statement that removes duplicates before the final INSERT ensures that only unique rows remain, resolving the duplication without altering the source data.

Answer

Enable the 'upsert' feature in the Redshift connection by setting 'update' to true.

Answer

Use partition pruning on the S3 source to reduce the number of files read.

Answer

Increase the number of DPUs (Data Processing Units) allocated to the job.

Question 3

A company is designing a data ingestion pipeline to load CSV files from an SFTP server into Amazon S3. The files are generated hourly and range from 10 MB to 500 MB. Which AWS service should be used to orchestrate the transfer with minimal operational overhead?

Accepted Answer

AWS Transfer Family. AWS Transfer Family is the correct choice because it provides a fully managed, serverless SFTP endpoint that can directly receive files from an SFTP server and automatically store them in Amazon S3. This eliminates the need to manage any compute infrastructure or write custom code for the transfer, minimizing operational overhead for hourly CSV file ingestion.

Answer

AWS Glue

Answer

Amazon AppFlow

Answer

AWS DataSync

Question 4

Arrange the steps to implement a data lake on Amazon S3 with AWS Lake Formation.

Accepted Answer

Create S3 bucket, then register it in Lake Formation, then set up administrators, then define schema in Glue Catalog, then grant access to users. Start by creating the S3 bucket. Then register it in Lake Formation, set up administrators, define the schema in Glue Catalog, and finally grant access to users.

Answer

Set up administrators, then create S3 bucket, then register it in Lake Formation, then define schema in Glue Catalog, then grant access to users

Answer

Create S3 bucket, then define schema in Glue Catalog, then register it in Lake Formation, then set up administrators, then grant access to users

Answer

Register S3 bucket in Lake Formation, then create S3 bucket, then set up administrators, then define schema in Glue Catalog, then grant access to users

Question 5

Order the steps to query data in Amazon Redshift Spectrum from an external table in Athena.

Accepted Answer

Create external schema in Redshift, then create external table, then grant permissions, then run the query, then verify results.. Start by creating the external schema in Redshift, then the external table, grant permissions, run the query, and verify results.

Answer

Run the query, then verify results, then create external table, then grant permissions, then create external schema.

Answer

Create external table, then create external schema, then grant permissions, then run the query, then verify results.

Answer

Create external schema, then create external table, then run the query, then grant permissions, then verify results.

Question 6

Match each AWS security service to its purpose in data protection.

Accepted Answer

AWS KMS: Manage encryption keys for data at rest and in transit. The correct matches are: AWS KMS for encryption key management, AWS IAM for access control, and AWS CloudTrail for API auditing. Common confusions include swapping these services' purposes.

Answer

AWS KMS: Control access to AWS resources through users, groups, and roles

Answer

AWS IAM: Manage encryption keys for data at rest and in transit

Answer

AWS CloudTrail: Manage encryption keys for data at rest and in transit

Question 7

A company is streaming clickstream data from a website into Amazon Kinesis Data Streams. The data is then consumed by a Lambda function that transforms the records and writes them to an S3 bucket in Parquet format. Recently, the Lambda function has been timing out and the S3 bucket is not receiving all expected records. The Kinesis stream has a shard count of 10 and the Lambda function's reserved concurrency is set to the default. Which change would MOST likely resolve the issue?

Accepted Answer

Decrease the batch window from the default 300 seconds to 60 seconds.. The Lambda function is timing out because it cannot process the default batch of 100 records within the function timeout. Decreasing the batch window from 300 seconds to 60 seconds causes Lambda to invoke more frequently with smaller batches, reducing the number of records per invocation. This lowers the processing time per invocation, helping the function complete before the timeout. Increasing the batch size (Option D) would worsen the issue by adding more records per invocation. Increasing reserved concurrency (Option C) does not reduce per-invocation processing time. Using a delivery stream (Option B) changes the architecture unnecessarily and may not preserve the transformation logic.

Answer

Configure the Kinesis stream to directly write to S3 using a delivery stream.

Answer

Increase the Lambda function's reserved concurrency.

Answer

Increase the batch size from the default 100 to 500 records per invocation.

Question 8

A data engineer is using AWS Glue to perform ETL on data stored in an S3 bucket. The source data is in CSV format with a header row, and the target is a set of Parquet files partitioned by date. The engineer notices that the Glue job is reading all files in the source prefix, including temporary files that should be ignored. What is the MOST efficient way to exclude these temporary files?

Accepted Answer

Use an S3 prefix exclusion pattern in the Glue job's source path.. AWS Glue supports S3 path exclusion patterns using glob-style syntax (e.g., `--exclude` or `excludePatterns` in the job parameters). By specifying a pattern like `**/_temporary/**` or `*.tmp`, the Glue job will skip those files during the read phase, avoiding unnecessary data processing and reducing costs. This is the most efficient approach as it requires no additional infrastructure or data movement.

Answer

Change the source format from CSV to Parquet.

Answer

Set up an S3 event notification to trigger a Lambda function that moves temporary files.

Answer

Create a custom classifier in the Glue Data Catalog.

Question 9

A company uses Amazon Kinesis Data Firehose to ingest JSON logs from multiple sources into an S3 data lake. The data is then consumed by Amazon Athena for analysis. Recently, some queries have been failing with the error 'HIVE_BAD_DATA: Field xyz's type is an unsupported type'. The firehose delivery stream transforms the data using a Lambda function that converts timestamps to Unix epoch. What is the MOST likely cause of the query failure?

Accepted Answer

Some records contain timestamps that were not converted to epoch, so Athena infers the column as a string.. The error 'HIVE_BAD_DATA: Field xyz's type is an unsupported type' occurs when Athena's schema inference encounters inconsistent data types for the same column. Since the Lambda function converts timestamps to Unix epoch, but some records may have failed conversion (e.g., due to malformed input or Lambda errors), those records retain the original string timestamp. Athena then sees a mix of numeric epoch values and string timestamps, causing it to infer the column as a string type, which is unsupported for the expected numeric operations in the query.

Answer

The data is in JSON format instead of Parquet.

Answer

The S3 partitions are not registered in the Glue Data Catalog.

Answer

The IAM role for Firehose does not have permission to write to S3.

Question 10

A company uses Amazon S3 to store raw data and AWS Glue to run ETL jobs that transform the data into analytics-ready tables. The Glue job reads from a source with a schema that changes frequently (new columns added). The engineer wants the Glue job to automatically adapt to schema changes without manual intervention. Which configuration should the engineer use?

Accepted Answer

Enable the 'Update schema' option in the Glue job's output target configuration.. Enabling the 'Update schema' option in the Glue job's output target configuration allows the job to automatically add new columns to the target table in the Data Catalog when the source schema changes. This setting directly addresses the requirement for automatic adaptation to schema changes without manual intervention, as it updates the table definition during the ETL job run.

Answer

Schedule a Glue crawler to run after each ETL job to update the Data Catalog.

Answer

Set the job to use schema-on-read by storing data in Parquet format.

Answer

Use Glue's partition indexes to automatically detect new columns.

Question 11

A company is running a critical application that generates millions of small JSON files every hour in an S3 bucket. A data engineer needs to process these files in near real-time using AWS Glue. The engineer wants to minimize the latency between file arrival and Glue job start. Which TWO actions should the engineer take?

Accepted Answer

Pre-process the files to consolidate them into larger files before the Glue job runs.. Option D is correct because S3 event notifications can be configured to invoke an AWS Lambda function on every object-created event, and that Lambda can call the Glue StartJobRun API immediately, eliminating the polling delay of Glue's native S3 crawler/trigger and giving the lowest possible latency between file arrival and job start. Option C is correct because Glue job startup and per-file overhead dominate when processing millions of tiny JSON files; consolidating them into larger files (for example, via an S3-triggered Lambda or a compaction step) reduces the number of input objects, the number of Spark tasks, and the scheduling/listing overhead, which directly shortens the time from arrival to effective processing. Option A is incorrect because increasing the batch window to 600 seconds deliberately delays job execution, which is the opposite of minimizing latency. Option B is incorrect because adding DPUs increases parallel processing throughput and may reduce total job runtime, but it does not reduce the latency between file arrival and job start, and it does not address the small-file overhead.

Answer

Increase the Glue job's batch window to 600 seconds.

Answer

Increase the number of DPUs for the Glue job to accelerate processing.

Question 12

A data engineer needs to ingest data from an Amazon RDS MySQL database into a data lake on Amazon S3. The engineer wants to perform an initial full load and then capture incremental changes. Which TWO AWS services can be combined to achieve this?

Accepted Answer

Amazon S3. AWS Database Migration Service (DMS) is correct because it can perform the initial full load from an Amazon RDS MySQL database and then continuously replicate ongoing changes using change data capture (CDC) from the MySQL binary log. Amazon S3 is correct because it serves as the target data lake storage where DMS lands the full-load and incremental change data, typically in CSV or Parquet files. Together, DMS plus S3 directly satisfy the requirement to ingest an initial full load and then capture incremental changes. Amazon Kinesis Data Firehose is not correct because it is a streaming delivery service, not a database migration or CDC tool for RDS MySQL. AWS Glue is not correct because, while it can perform ETL and crawl data, it does not natively provide continuous CDC replication from an RDS MySQL source. AWS Data Pipeline is not correct because it is a legacy orchestration service for scheduled data workflows and does not provide MySQL change data capture.

Answer

Amazon Kinesis Data Firehose

Answer

AWS Glue

Answer

AWS Data Pipeline

Question 13

A company is using Amazon Kinesis Data Streams to process real-time stock trade data. The data is consumed by a Lambda function that calculates moving averages and stores results in Amazon DynamoDB. The Lambda function is failing with 'ProvisionedThroughputExceededException' on the DynamoDB table. The table has on-demand capacity. Which TWO actions should the engineer take to resolve this issue?

Accepted Answer

Add a dead-letter queue and configure the Lambda function to retry on failure with exponential backoff.. Option A is correct because a ProvisionedThroughputExceededException from DynamoDB is a throttling error that can be transient; adding a dead-letter queue and configuring retries with exponential backoff lets the Lambda function absorb temporary throttling without losing Kinesis records, which is the recommended resilience pattern for stream consumers. Option D is correct because increasing the batch size of the Kinesis event source mapping makes each Lambda invocation write a larger number of items in fewer, larger DynamoDB operations, which reduces the number of individual write requests hitting the table and lowers the chance of exceeding the on-demand per-partition throughput limits. Option B is not appropriate because decreasing the batch window to 0 seconds causes Lambda to invoke more frequently with smaller batches, increasing the number of DynamoDB write calls and worsening throttling. Option C is not appropriate because reserved concurrency controls how many Lambda instances can run, but it does not increase DynamoDB on-demand capacity; higher concurrency would actually generate more concurrent writes and more throttling.

Answer

Decrease the batch window to 0 seconds to process records immediately.

Answer

Increase the Lambda function's reserved concurrency to process more shards.

Question 14

A data engineer is designing a data ingestion pipeline for IoT sensor data. The sensors send JSON messages every second, and the data must be stored in Amazon S3 in near real-time (within 5 minutes). The engineer also needs to transform the data by adding a timestamp and filtering out malformed records. Which THREE services should be used together?

Accepted Answer

AWS Glue. AWS IoT Core (D) is correct because it is the managed service designed to securely connect IoT sensors over MQTT/HTTPS and ingest their JSON messages at scale, which is the required entry point for the sensor data. Amazon Kinesis Data Firehose (E) is correct because it can consume that streaming data and deliver it into Amazon S3 in near real-time, satisfying the within-5-minute storage requirement with buffering intervals as low as 60 seconds. AWS Glue (A) is correct because it provides the ETL capability to transform the data by adding a timestamp and filtering out malformed records before or as the data lands in S3. Amazon Athena (B) is not appropriate here because it is an interactive query service for analyzing data already in S3, not an ingestion or transformation component of the pipeline. Amazon SQS (C) is not the right fit because it is a generic message queue for decoupling applications, not a purpose-built IoT ingestion or streaming delivery service to S3.

Answer

Amazon Athena

Answer

Amazon Simple Queue Service (SQS)

Question 15

The Glue job attempts to read data from 's3://my-data-bucket/input/' and write to 's3://my-data-bucket/output/'. It also tries to update a table in the Glue Data Catalog. The job fails with an access denied error. What is the MOST likely cause?

Accepted Answer

The IAM role is missing the 's3:ListBucket' permission on the bucket.. The Glue job reads from 's3://my-data-bucket/input/' and writes to 's3://my-data-bucket/output/'. For S3 read/write operations, the IAM role must have 's3:ListBucket' permission on the bucket itself (my-data-bucket) to allow listing of objects, in addition to 's3:GetObject' and 's3:PutObject' on the object ARN. Without 's3:ListBucket', the job cannot enumerate objects in the input prefix, leading to an access denied error.

Answer

The 'glue:UpdateTable' action is not allowed on the specific table.

Answer

The policy is missing a condition key for the S3 bucket.

Answer

The resource ARN does not include the bucket itself; it only covers objects.