DEA-C01 Data Ingestion and Transformation Practice Test 29 — 15 Questions

Question 1

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an Amazon S3 bucket. The data is delivered in JSON format. The company wants to convert the data to Apache Parquet format before delivery to reduce storage costs and improve query performance. How can this be achieved?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use the AWS Glue Data Catalog to define a schema and configure Firehose to use it for Parquet conversion.. Kinesis Data Firehose can convert the input data to Parquet or ORC format using a schema from the AWS Glue Data Catalog. Option A is incorrect because delivering as JSON then converting with Athena is an extra step after storage, not before delivery, and does not reduce storage costs from the outset. Option C is incorrect because Lambda can be used for custom transformations, but Firehose natively supports Parquet conversion using Glue. Option D is incorrect because Firehose cannot convert to Parquet automatically without a schema; you must provide a schema (e.g., from Glue).

Answer

Deliver data to S3 as JSON, then use Amazon Athena to convert to Parquet.

Answer

Write an AWS Lambda function to transform the data to Parquet and deliver it to S3.

Answer

Configure the Firehose stream to convert data to Parquet automatically without any additional setup.

Question 2

A company is using AWS Glue to process data from an Amazon S3 data lake. The Glue job runs daily and transforms data into multiple output formats. Which TWO actions can the company take to optimize the Glue job's performance and reduce costs? (Choose TWO.)

Accepted Answer

Increase the number of DPUs allocated to the job.. Options A and E are correct. Increasing the number of DPUs (A) can parallelize processing and improve performance for data-intensive Glue jobs, potentially reducing runtime and cost if the job runs faster. Enabling job bookmarking (E) allows Glue to track processed data and process only new or changed data in incremental runs, reducing processing time and cost by avoiding full reprocessing. Option B (reducing DPUs) would likely decrease performance. Option C (disabling bookmarking) would force full reprocessing, increasing cost and time. Option D (increasing job timeout) does not optimize performance or cost; it only allows the job to run longer, which could increase cost if the job is inefficient.

Answer

Reduce the number of DPUs (Data Processing Units) allocated to the job.

Answer

Disable job bookmarking to force full reprocessing every run.

Answer

Increase the job timeout to allow more time for processing.

Question 3

A company is running a 10-node Amazon EMR cluster to process data from Amazon S3. The cluster is using Apache Spark for transformations. The data processing is taking longer than expected. Which THREE actions can improve the performance of the Spark jobs on EMR? (Choose THREE.)

Accepted Answer

Enable dynamic allocation of executors.. Enabling dynamic allocation of executors allows Amazon EMR to automatically scale the number of executors up or down based on workload demand. This prevents resource underutilization or over-provisioning, which can significantly improve Spark job performance by ensuring that the cluster's resources are efficiently matched to the processing needs of the transformations.

Answer

Reduce the number of shuffle partitions.

Answer

Disable speculative execution to reduce redundant tasks.

Question 4

A company is using AWS Glue to catalog data in Amazon S3. The data is stored in CSV format, but the schema is not consistent across all files. Which TWO actions can the company take to handle schema evolution and ensure the Glue Data Catalog is up to date? (Choose TWO.)

Accepted Answer

Configure the Glue crawler to update the table schema on each run.. Options A and D are correct. Configuring the Glue crawler to update the table schema on each run (A) allows automatic schema evolution, while scheduling the crawler to run periodically (D) ensures that changes in the data are captured. Option B (manual update) is not scalable. Option C (disabling schema update) would prevent automatic updates. Option E (fixed schema) is impractical for evolving data.

Answer

Manually update the Glue Data Catalog tables whenever the schema changes.

Answer

Disable schema update in the crawler and add partitions manually.

Answer

Require all data producers to use a single fixed schema.

Question 5

A data engineer is reviewing the S3 Lifecycle policy for a data lake bucket. The goal is to archive log data after 30 days and delete it after 365 days, and delete temporary data after 1 day. What is wrong with the current configuration?

Accepted Answer

The transition to GLACIER should be after 30 days, but the expiration should be after 365 days from the transition, not from creation.. The current lifecycle configuration is correct: it transitions objects to GLACIER after 30 days and expires them 365 days after creation. However, a common mistake is to configure expiration relative to the transition date. Option D identifies this error by stating that expiration should be based on creation date, not transition. The other options are incorrect: A - prefix filters do not require wildcards; B - expiration actions do not need a prior transition; C - S3 Lifecycle can expire GLACIER objects without restoration.

Answer

The prefix filter for the first rule does not include a wildcard, so it may not match all log files.

Answer

The rule for temp data has no transition, so it will not expire objects.

Answer

The expiration for the first rule will not delete objects in GLACIER storage class unless they are restored first.

Question 6

A data engineer is troubleshooting an AWS Glue job that is failing with an Access Denied error when trying to read data from an S3 bucket. The IAM policy attached to the Glue job's IAM role is shown in the exhibit. What is the likely cause of the failure?

Accepted Answer

The policy does not include s3:ListBucket permission on the bucket.. The IAM policy grants s3:GetObject and s3:PutObject on the bucket objects, but it does not include s3:ListBucket permission on the bucket itself. When AWS Glue reads data from S3, it needs to list the objects in the bucket first (s3:ListBucket) to discover which objects to read. Without this permission, the Glue job fails with an Access Denied error even though it has GetObject permission. Option A is incorrect because the policy does include s3:GetObject (implied for objects) and the issue is not about PutObject. Option B is incorrect because glue:StartJobRun is not related to S3 access; the error is S3-specific. Option D is incorrect because glue:GetJobRun is unrelated to the S3 access issue.

Answer

The policy does not include s3:GetObject or s3:PutObject permissions.

Answer

The policy does not include glue:StartJobRun permission.

Answer

The policy does not include glue:GetJobRun permission.

Question 7

A data engineer is troubleshooting a Kinesis Data Streams application that is experiencing high latency. The stream has 2 shards. The application is using a single Kinesis Client Library (KCL) worker to process all shards. Which change will MOST likely reduce latency?

Accepted Answer

Deploy multiple KCL workers to process shards in parallel.. The application uses a single KCL worker to process all 2 shards, which processes records sequentially and causes high latency. Deploying multiple KCL workers (ideally one per shard) enables parallel processing of shards, significantly reducing latency. Option A is incorrect because increasing shard count to 4 adds more capacity but does not address the bottleneck of a single worker; the same worker would process all 4 shards sequentially, potentially worsening latency. Option C is incorrect because Kinesis Data Streams is a managed service; there is no instance type to change for the stream itself. The KCL worker runs on your compute resources, not on the stream. Option D is incorrect because decreasing shards to 1 reduces the level of parallelism, increasing the workload per shard and likely increasing latency further.

Answer

Increase the number of shards to 4.

Answer

Use a larger instance type for the Kinesis stream.

Answer

Decrease the number of shards to 1.

Question 8

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an Amazon S3 bucket. The data is delivered in JSON format. The company wants to convert the data to Apache Parquet format before delivery to reduce storage costs and improve query performance. How can this be achieved?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Reduce the buffer interval from 300 seconds to 60 seconds.. The buffer interval determines how often data is delivered to the destination. Changing it from 300 seconds to 60 seconds will deliver data every minute. Option B (buffer size) affects delivery based on data volume, not time. Option C (compression) does not affect frequency. Option D (partitioning) does not affect delivery frequency.

Answer

Increase the buffer size to trigger delivery sooner.

Answer

Enable dynamic partitioning to deliver data more frequently.

Answer

Enable compression to reduce data size and speed up delivery.

Question 9

A data engineering team needs to ingest real-time streaming data from thousands of IoT devices and transform the data before storing it in Amazon S3. Which AWS service is most suitable for performing the transformation step in near real-time?

Accepted Answer

AWS Lambda. AWS Lambda can run code in response to Kinesis Data Streams events and perform transformations before writing to S3. Option B is incorrect because Kinesis Data Analytics is for running SQL/Java on streams, not simple transforms. Option C is incorrect because Kinesis Data Firehose is for loading data to destinations with optional simple transformations via Lambda. Option D is incorrect because Amazon S3 is storage, not a transformation service.

Answer

Amazon Kinesis Data Analytics

Answer

Amazon Kinesis Data Firehose

Answer

Amazon S3

Question 10

A company uses Amazon Kinesis Data Firehose to ingest application logs into an Amazon S3 bucket. The logs are in JSON format. The data engineering team wants to convert the logs from JSON to Parquet format before landing in S3. What is the most cost-effective way to achieve this?

Accepted Answer

Configure the Firehose delivery stream to convert the data to Parquet using a schema from AWS Glue.. Kinesis Data Firehose can convert the input data format to Parquet using a schema from AWS Glue. Option A is incorrect because Athena queries raw data and would not help with ingestion. Option C is incorrect because Lambda can do this but would incur additional compute cost. Option D is incorrect because Glue ETL would add cost and latency.

Answer

Use Amazon Athena to query the JSON data and write results in Parquet format.

Answer

Use an AWS Lambda function to transform each record to Parquet and send to Firehose.

Answer

Use an AWS Glue ETL job to run on a schedule and convert JSON to Parquet in S3.

Question 11

A company is ingesting streaming data from social media feeds using Amazon Kinesis Data Streams. The data volume peaks at 10,000 records per second, and each record is up to 1 KB. The company needs to archive the raw data in Amazon S3 in near real-time and also make it available for real-time analytics using Amazon Kinesis Data Analytics. What is the MOST efficient architecture to meet these requirements?

Accepted Answer

Use Kinesis Data Streams as the ingestion point. Use Kinesis Data Firehose to read from the stream and write to S3. Use Kinesis Data Analytics to read directly from the same stream.. Kinesis Data Streams can serve as a single ingestion point, with Kinesis Data Firehose reading from the stream to deliver data to S3 (with optional transformation) and Kinesis Data Analytics reading directly from the same stream for real-time analytics. This avoids unnecessary duplication of streams or Lambda-based processing, which would add latency and complexity. The architecture is the most efficient as it leverages native integrations without intermediate compute.

Answer

Use Kinesis Data Streams as the ingestion point. Use Kinesis Data Firehose to read from the stream, convert to Parquet, and write to S3. Use a Lambda function to send data to Kinesis Data Analytics.

Answer

Use Kinesis Data Streams as the ingestion point. Use a Lambda function to read from the stream, write to S3, and send data to Kinesis Data Analytics.

Answer

Use two Kinesis Data Streams: one for S3 delivery and one for Kinesis Data Analytics.

Question 12

A data engineer needs to design a data ingestion pipeline that ingests CSV files from an Amazon S3 bucket, transforms the data by adding a timestamp column, and loads it into an Amazon Redshift table. The pipeline should run automatically whenever a new file is uploaded to the S3 bucket. Which AWS service should be used to trigger the transformation?

Accepted Answer

AWS Lambda. Amazon S3 can be configured to send events directly to AWS Lambda when a new CSV file is uploaded. Lambda then executes the transformation (adding a timestamp column) and loads the data into Redshift. Option B (Step Functions) is not triggered directly by S3 events without an intermediate service like Lambda. Option C (EventBridge) can route S3 events to Lambda, but a direct S3 event notification to Lambda is simpler and more common; however, the key point is that Lambda is the direct trigger. Option D (SQS) requires a separate process to poll the queue and invoke Lambda; it is not a direct trigger.

Answer

AWS Step Functions

Answer

Amazon EventBridge

Answer

Amazon Simple Queue Service (SQS)

Question 13

A company uses Amazon Kinesis Data Firehose to deliver data to an Amazon S3 bucket. The data is in JSON format and contains a 'timestamp' field with a Unix epoch value. The company wants to partition the S3 objects by year, month, day, and hour based on the timestamp. What is the MOST efficient method to achieve this?

Accepted Answer

Use the dynamic partitioning feature of Kinesis Data Firehose with inline parsing to extract the timestamp and create the S3 prefix.. Kinesis Data Firehose supports dynamic partitioning with inline parsing to extract the timestamp and create the S3 prefix by year, month, day, and hour. Option B is incorrect because a custom prefix based on current time would use the delivery time, not the event timestamp, so partitioning would not reflect the actual data timestamps. Option C is incorrect because using an AWS Glue ETL job introduces additional latency and complexity; Firehose can partition directly without needing an extra service. Option D is incorrect because Amazon Athena is a query engine, not an ingestion tool; running a CTAS query would require the data to already be in S3 and adds overhead.

Answer

Configure a custom S3 prefix in Firehose using the 'YYYY/MM/dd/HH' format based on the current time.

Answer

Use an AWS Glue ETL job to read from Firehose, partition, and write to S3.

Answer

Use Amazon Athena to run a CTAS query that partitions the data by timestamp.

Question 14

A company needs to ingest data from a relational database into Amazon S3 for analytics. The database is an Amazon RDS MySQL instance. Which AWS service should be used for a one-time historical data load?

Accepted Answer

AWS Database Migration Service (DMS). AWS Database Migration Service (DMS) is the correct choice for a one-time historical data load from Amazon RDS MySQL to Amazon S3. DMS supports full-load migrations from relational databases to S3, making it ideal for this use case. AWS Glue ETL can also perform similar tasks but is more suited for complex transformations and scheduled jobs, and DMS is the dedicated service for database migrations. Amazon Athena is a query service, not an ingestion tool. Amazon Kinesis Data Firehose is designed for streaming data, not one-time loads.

Answer

AWS Glue ETL

Answer

Amazon Athena

Answer

Amazon Kinesis Data Firehose

Question 15

A company is using Amazon Kinesis Data Streams with a Lambda consumer to process clickstream data. The data rate is high and the Lambda function is falling behind, resulting in increased processing latency. What is the MOST effective way to improve throughput?

Accepted Answer

Increase the number of shards in the Kinesis stream.. Increasing the number of shards in the Kinesis stream increases the stream's total read capacity, allowing more concurrent Lambda invocations to process records in parallel. Since each shard supports up to 5 read transactions per second and a maximum of 2 MB/s read throughput, adding shards directly raises the aggregate throughput, enabling the Lambda consumer to keep up with the high data rate.

Answer

Increase the memory allocated to the Lambda function.

Answer

Increase the Lambda function timeout.

Answer

Use Kinesis Data Firehose instead of Lambda.