DEA-C01 Data Ingestion and Transformation Practice Test 29 — 15 Questions

Question 1

A company uses AWS Glue to process streaming data from Amazon Kinesis Data Streams. The data is JSON formatted and includes a timestamp field. The company wants to partition the output in Amazon S3 by date and hour, and ensure exactly-once processing semantics. Which combination of configurations should be used?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Enable checkpointing in the AWS Glue streaming job and specify an S3 location for checkpoint data.. AWS Glue streaming jobs require checkpointing to track the progress of data consumption from Kinesis Data Streams and to ensure exactly-once processing semantics. By enabling checkpointing and specifying an S3 location, Glue periodically saves the state of processed records, allowing it to resume from the last committed offset in case of failures, thus preventing duplicates or data loss.

Answer

Disable checkpointing and use the 'exactly_once' delivery option in Kinesis Data Streams.

Answer

Use Amazon DynamoDB as a checkpoint store by configuring the Glue job with a DynamoDB connection.

Answer

Use Kinesis Client Library (KCL) checkpointing with a DynamoDB table.

Question 2

A company uses Amazon Kinesis Data Firehose to ingest log data from web servers into Amazon S3. The data is in JSON format and each record is approximately 2 KB. The delivery stream is configured to buffer incoming records for 60 seconds or 5 MB, whichever comes first. The company notices that the data in S3 is delayed by up to 5 minutes during peak hours. Which action would most effectively reduce the delivery latency?

Accepted Answer

Decrease the buffer interval to 15 seconds.. The observed delay of up to 5 minutes during peak hours indicates that the buffer size threshold (5 MB) is rarely reached because each record is only ~2 KB, so the delivery stream relies on the buffer interval (60 seconds) to trigger delivery. By decreasing the buffer interval to 15 seconds, Kinesis Data Firehose will push data to S3 more frequently, directly reducing the maximum latency from 60 seconds to 15 seconds per batch, which eliminates the compounding delays caused by queuing during high-throughput periods.

Answer

Increase the buffer size to 10 MB to allow more records per delivery.

Answer

Enable compression (GZIP) on the delivery stream.

Answer

Enable data transformation with AWS Lambda to convert JSON to Parquet.

Question 3

A company is ingesting real-time clickstream data into Amazon S3 using Amazon Kinesis Data Firehose. The data is semi-structured and the company wants to transform the data into Parquet format and partition it by year, month, day, and hour. Which TWO steps should be taken to achieve this? (Choose TWO.)

Accepted Answer

Enable dynamic partitioning in Kinesis Data Firehose and specify the partition keys as year, month, day, hour extracted from the data.. Kinesis Data Firehose's dynamic partitioning feature allows you to specify partition keys (year, month, day, hour) extracted from the incoming data, and Firehose will automatically create the corresponding S3 prefix structure (e.g., year=2024/month=01/day=15/hour=10/) during delivery. Option D is correct because to convert semi-structured data to Parquet format, you can attach an AWS Lambda function as a data transformation to Firehose, which converts each record to Parquet before delivery to S3. Option E is incorrect because while Kinesis Data Firehose does support converting data to Parquet format using a schema from the AWS Glue Data Catalog, this approach requires a pre-defined Glue schema and is less flexible for semi-structured data. Moreover, the question does not mention any existing Glue Data Catalog, and the Lambda transformation in option D is a more direct and customizable method for the transformation needed.

Answer

Set up an Amazon S3 event notification to trigger an AWS Lambda function that partitions the data after delivery.

Answer

Use an AWS Glue Crawler to infer the schema and automatically partition the data in S3.

Answer

Configure Kinesis Data Firehose to convert the data to Parquet format using a schema from the AWS Glue Data Catalog.

Question 4

Arrange the steps to create an AWS Glue job that transforms data from Amazon S3 to Amazon Redshift in the correct order.

Accepted Answer

Step 1: Catalog the source data with a crawler. Step 2: Prepare the ETL script. Step 3: Configure the job with connections. Step 4: Run the job. Step 5: Verify the results in Amazon Redshift.. First, catalog the source data with a crawler. Then, prepare the ETL script. Configure the job with connections, run it, and finally verify the results in Redshift.

Answer

Step 1: Prepare the ETL script. Step 2: Catalog the source data with a crawler. Step 3: Configure the job with connections. Step 4: Run the job. Step 5: Verify the results in Amazon Redshift.

Answer

Step 1: Configure the job with connections. Step 2: Catalog the source data with a crawler. Step 3: Prepare the ETL script. Step 4: Run the job. Step 5: Verify the results in Amazon Redshift.

Answer

Step 1: Catalog the source data with a crawler. Step 2: Configure the job with connections. Step 3: Prepare the ETL script. Step 4: Run the job. Step 5: Verify the results in Amazon Redshift.

Question 5

Order the steps to troubleshoot a failed AWS Glue job that reads from JDBC and writes to S3.

Accepted Answer

Check AWS Glue job logs, then verify network connectivity, then verify IAM permissions, then test JDBC connection, then review the Glue script.. Start with logs to identify errors, then check connectivity, IAM permissions, test connection, and review script.

Answer

Verify IAM permissions, then check AWS Glue job logs, then verify network connectivity, then test JDBC connection, then review the Glue script.

Answer

Test JDBC connection, then check AWS Glue job logs, then verify network connectivity, then verify IAM permissions, then review the Glue script.

Answer

Review the Glue script, then check AWS Glue job logs, then verify network connectivity, then verify IAM permissions, then test JDBC connection.

Question 6

Match each AWS service to its primary purpose in data engineering.

Accepted Answer

Amazon S3: Object storage for data lakes. The correct matches are: Amazon S3 → object storage for data lakes (A), AWS Glue → serverless ETL (B), Amazon Athena → interactive SQL queries on S3 (C), and Amazon Redshift → data warehousing (D). Common confusions involve mixing up services like Kinesis (streaming) with Glue (ETL) or Data Pipeline (orchestration) with Kinesis (ingestion).

Answer

AWS Kinesis: Serverless ETL service

Answer

AWS Data Pipeline: Real-time data ingestion

Question 7

Match each AWS data migration tool to its primary function.

Accepted Answer

AWS Database Migration Service (DMS): Migrates databases to AWS with minimal downtime. AWS DMS migrates databases with minimal downtime; AWS Snowball transfers large datasets physically; AWS DataSync automates online data transfer. Common confusions include swapping tool functions or attributing schema conversion to DataSync instead of SCT.

Answer

AWS Database Migration Service (DMS): Transfers large datasets via physical devices

Answer

AWS Snowball: Migrates databases to AWS with minimal downtime

Answer

AWS DataSync: Converts database schema to target platform

Question 8

Match each AWS database service to its primary use case.

Accepted Answer

Amazon RDS: Relational database for online transaction processing (OLTP) workloads.. Correct matches: RDS -> OLTP relational, DynamoDB -> NoSQL low-latency, Redshift -> data warehousing, ElastiCache -> in-memory caching. Common confusions include swapping RDS and DynamoDB use cases.

Answer

Amazon RDS: NoSQL key-value database for low-latency applications.

Answer

Amazon DynamoDB: Relational database for OLTP workloads.

Question 9

A data engineer is designing a data ingestion pipeline to load data from an on-premises Oracle database to Amazon S3. The pipeline should capture changes in near real-time (within minutes) and minimize impact on the source database. The source table has a 'last_modified' timestamp column. Which service combination would meet these requirements?

Accepted Answer

AWS DMS with a replication task in CDC mode, writing to S3 in Parquet format.. AWS DMS with a replication task in CDC (Change Data Capture) mode is the correct choice because it continuously reads the Oracle redo logs to capture near real-time changes (within seconds to minutes) with minimal impact on the source database. It can directly write to S3 in Parquet format, meeting the requirement for low-latency ingestion without full table scans.

Answer

Amazon Kinesis Data Firehose with a Lambda function that queries Oracle.

Answer

AWS Data Pipeline with a periodic SQL query activity to copy full table snapshots.

Answer

AWS Glue with a JDBC connection to Oracle, running a crawler every 5 minutes.

Question 10

A company uses AWS Glue to process data from multiple S3 buckets. The Glue job runs daily and reads data from a bucket that contains millions of small files (each < 1 MB). The job has been running for hours and is often close to the 8-hour timeout limit. Which optimization would MOST reduce the job's runtime?

Accepted Answer

Pre-process the data to consolidate small files into larger files before the Glue job.. Consolidating millions of small files (<1 MB each) into larger files before the Glue job is the most impactful optimization because Spark and Glue incur massive per-file overhead: each file requires a separate S3 GET, task scheduling, and metadata operation. With millions of tiny files, the job spends most of its time on I/O and task overhead rather than actual processing. Merging them into a few hundred MB files dramatically reduces task count and runtime.

Answer

Convert the source data from CSV to Parquet format.

Answer

Increase the number of DPUs allocated to the Glue job.

Answer

Use a larger Spark shuffle partition size.

Question 11

A data engineer is setting up an Amazon Kinesis Data Firehose delivery stream to load data into Amazon Redshift. The data is coming from an application that produces JSON records. The engineer needs to transform the data to match the Redshift table schema. Which approach is the MOST cost-effective and requires the least operational overhead?

Accepted Answer

Configure a Lambda function in the Firehose delivery stream to transform records before delivery.. Kinesis Data Firehose natively supports invoking a Lambda function as a transformation step within the delivery stream. This allows the engineer to write a simple Lambda function that parses the incoming JSON records and transforms them to match the Redshift table schema, all without provisioning or managing any additional infrastructure. This approach is the most cost-effective (pay per invocation) and requires the least operational overhead since Firehose handles the orchestration, retries, and delivery to Redshift automatically.

Answer

Use AWS Glue as a transformation step between Firehose and Redshift, with a trigger on S3.

Answer

Use Kinesis Data Firehose with direct PUT to Redshift and rely on Redshift's COPY command to transform.

Answer

Use the Kinesis Client Library (KCL) to consume the stream, transform in an EC2 instance, and then load to Redshift.

Question 12

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a fleet of EC2 instances running a custom application that processes the records and writes to DynamoDB. The application is experiencing high latency and records are being processed slower than they are produced. The stream has 5 shards. Which action would MOST effectively improve processing speed?

Accepted Answer

Increase the number of shards in the Kinesis stream.. The bottleneck is the number of shards in the Kinesis stream. Each shard provides a fixed read capacity of 2 MB/s and 5 read transactions per second. With only 5 shards, the total read throughput is limited regardless of how many EC2 instances consume the data. Increasing the number of shards increases the total read capacity, allowing more records to be consumed in parallel and reducing processing latency.

Answer

Use the Kinesis Client Library (KCL) to automatically distribute shards among instances.

Answer

Increase the EC2 instance size to provide more CPU and memory.

Answer

Add more EC2 instances consuming from the same stream without changing shard count.

Question 13

A company is building a data lake on Amazon S3 and wants to ingest data from multiple AWS services (CloudTrail, VPC Flow Logs, and ALB logs). The data should be stored in a central S3 bucket with a common partitioning scheme. Which service can be used to collect and centralize this data with minimal configuration?

Accepted Answer

Configure each source service to deliver logs directly to the central S3 bucket.. CloudTrail, VPC Flow Logs, and ALB logs can each be configured to deliver logs directly to a specified S3 bucket, including a central bucket, with no intermediary service required. This approach minimizes configuration overhead and avoids data movement costs, as each service writes natively to S3 using its own built-in delivery mechanism.

Answer

Use AWS Data Pipeline to copy logs from each source S3 bucket to the central bucket.

Answer

Use AWS Glue to crawl the logs from each source and write to a central S3 bucket.

Answer

Set up Amazon Kinesis Data Firehose to ingest logs from each service and write to S3.

Question 14

A company uses AWS Glue ETL jobs to transform data stored in Amazon S3. The job reads data in Parquet format, applies transformations, and writes the output back to S3 in Parquet format. The team wants to improve the job's performance and reduce costs. Which action is MOST effective?

Accepted Answer

Use column pruning and predicate pushdown to read only necessary columns and filter data early.. Column pruning and predicate pushdown reduce the amount of data read from S3 by Spark-based AWS Glue ETL jobs. By reading only the necessary columns and filtering rows early in the scan, I/O and memory usage decrease, directly improving performance and reducing costs.

Answer

Change the input format from Parquet to CSV to simplify parsing.

Answer

Coalesce the input data into a single large file before processing.

Answer

Increase the number of workers to maximum allowed.

Question 15

A data engineering team is building a data lake on Amazon S3. They need to ingest data from multiple sources: (1) streaming IoT data, (2) daily CSV exports from an on-premises system via SFTP, and (3) change data capture (CDC) from an Amazon Aurora database. Which THREE services should the team use to ingest these data sources?

Accepted Answer

Amazon Kinesis Data Streams for IoT data ingestion.. Amazon Kinesis Data Streams (A) is the right choice for streaming IoT data because it is a fully managed, scalable service designed for real-time ingestion of high-volume streaming records into S3 via Kinesis Data Firehose or consumers. AWS Database Migration Service (B) supports ongoing change data capture (CDC) from Amazon Aurora (and other relational databases), replicating inserts, updates, and deletes continuously, which matches the CDC requirement. AWS Transfer Family (C) provides a fully managed SFTP endpoint backed by Amazon S3, so the on-premises system can push daily CSV exports over SFTP directly into the data lake. AWS Glue ETL (D) is a batch/transform service, not a CDC ingestion mechanism, so it does not satisfy the Aurora CDC requirement. Amazon EMR (E) is a big data processing cluster, not a managed ingestion service for daily SFTP CSV files, so it is not the appropriate ingestion tool here.

Answer

AWS Glue ETL for CDC from Aurora.

Answer

Amazon EMR for daily CSV ingestion.