DEA-C01 Data Ingestion and Transformation Practice Test 47 — 15 Questions

Question 1

A financial services company is building a real-time fraud detection system. Transaction data is ingested via Amazon Kinesis Data Streams and processed by an Amazon Kinesis Data Analytics for Apache Flink application that runs sliding window aggregations. The output is written to an Amazon S3 bucket for downstream analysis. The Flink application is configured with parallelism of 4 and checkpointing every minute. The company has noticed that the application is experiencing high latency and the checkpointing is frequently failing. The CloudWatch metrics show that the Flink application's CPU utilization is near 100% and the checkpoint duration is spiking to over 5 minutes. The data engineer needs to improve performance. Which action should the data engineer take?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the parallelism of the Flink application to distribute the workload across more resources.. The Flink application is experiencing high CPU utilization and checkpointing failures, indicating that the current parallelism of 4 is insufficient to handle the workload. Increasing parallelism distributes the workload across more resources, reducing CPU load per task and allowing checkpointing to complete faster. This is a standard scaling approach for Flink applications in Kinesis Data Analytics.

Answer

Increase the number of shards in the source Kinesis stream to improve throughput.

Answer

Increase the heap memory of the Flink application to handle larger state.

Answer

Decrease the checkpoint interval to 30 seconds to reduce the amount of state being checkpointed.

Question 2

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data from a website. The data is consumed by an Amazon Kinesis Data Analytics for Apache Flink application that performs real-time analytics. The Flink application writes its results to an Amazon S3 bucket. The company has noticed that the Flink application is experiencing high checkpoint failure rates, causing delays. The CloudWatch metrics show that the checkpoint size is large and increasing. The data engineer needs to reduce the checkpoint size. Which action should the data engineer take?

Accepted Answer

Enable incremental checkpointing in the Flink application to only write changes since the last checkpoint.. Enabling incremental checkpointing in the Flink application is the correct action because it only writes the changes since the last checkpoint, significantly reducing checkpoint size. This is especially effective when state is large and growing, as it avoids rewriting the entire state with each checkpoint.

Answer

Decrease the checkpoint interval to reduce the amount of state accumulated.

Answer

Reduce the parallelism of the Flink application.

Answer

Increase the state time-to-live (TTL) configuration to retain state longer.

Question 3

A healthcare company is ingesting patient data from a legacy system into an Amazon S3 data lake using AWS Glue. The legacy system produces CSV files with inconsistent schemas (columns may appear or disappear in different files). The data engineer needs to create a Glue ETL job that can handle schema evolution and transform the data into a standardized parquet format. The job should also be able to process new files as they arrive. Which approach should the data engineer use?

Accepted Answer

Use AWS Glue DynamicFrames to read the CSV files and apply transformations using resolveChoice and applyMapping.. AWS Glue DynamicFrames are designed to handle schema evolution and inconsistent data. Using resolveChoice to handle columns that appear/disappear and applyMapping to standardize the schema allows the job to process files with varying schemas and output consistent Parquet.

Answer

Use AWS Glue crawlers to create a schema in the Data Catalog and then use a standard Spark DataFrame for transformation.

Answer

Use a Python shell job in Glue to manually parse each file and write to parquet.

Answer

Use a Glue ETL job with a static schema defined in the script and ignore files that don't match.

Question 4

A data engineering team is using AWS DMS to migrate a 2 TB Oracle database to Amazon RDS for PostgreSQL. The migration must have minimal downtime and needs to capture ongoing changes after the full load. Which THREE resources are required for this task? (Choose three.)

Accepted Answer

A DMS source endpoint configured for Oracle.. Option A is correct because AWS DMS requires a source endpoint that defines the Oracle database connection details, including server name, port, SID/service name, and credentials, so the replication instance can read the full load data and ongoing redo/archive log changes. Option B is correct because the AWS DMS replication instance is the managed compute resource that actually runs the migration tasks, performs the full load, and applies change data capture (CDC) from Oracle to the target. Option E is correct because a target endpoint pointing to the Amazon RDS for PostgreSQL instance is mandatory so DMS knows where to write the migrated tables and replicated changes. Option C is not needed because Snowball Edge is an offline physical transfer device, whereas DMS performs the migration over the network and supports ongoing replication. Option D is not required because DMS does not need an Amazon S3 staging bucket for an Oracle-to-RDS PostgreSQL migration; S3 is only used for certain source/target types such as S3 endpoints or specific native backup/restore workflows.

Answer

An AWS Snowball Edge device for initial data transfer.

Answer

An Amazon S3 bucket for staging the data.

Question 5

An e-commerce company is building a near-real-time dashboard to monitor customer clickstream data. The data is ingested via Amazon Kinesis Data Streams, transformed using AWS Lambda, and stored in Amazon S3. The team needs to query the data using Amazon Athena. Which THREE steps should be taken to optimize cost and performance? (Choose three.)

Accepted Answer

Convert the data to Apache Parquet or ORC format.. Option C is correct because converting the clickstream data to columnar formats like Apache Parquet or ORC lets Athena read only the columns referenced in each query and applies columnar compression and predicate pushdown, dramatically reducing bytes scanned and therefore cost and latency. Option D is correct because compressing data with gzip or snappy reduces the amount of data Athena must read from S3, lowering query cost and improving performance; snappy is especially effective with Parquet/ORC since it is splittable and column-oriented. Option E is correct because partitioning the S3 data by date (e.g., year/month/day) enables Athena partition pruning so queries that filter on time only scan the relevant prefixes instead of the entire dataset. Option A is not among the marked answers: while the AWS Glue Data Catalog is the standard metastore Athena uses, defining table metadata is a prerequisite for querying rather than a cost/performance optimization step for the data itself. Option B is not correct because storing data as JSON is row-oriented and non-columnar, which forces Athena to scan and parse entire records, increasing bytes scanned and query cost compared with Parquet or ORC.

Answer

Use AWS Glue Data Catalog to store the table metadata.

Answer

Store the data in JSON format for flexibility.

Question 6

A data engineering team uses AWS Glue to extract, transform, and load (ETL) data from Amazon RDS for MySQL to Amazon S3. The job runs daily and processes incremental data. The team notices that the job is taking longer than expected. Which TWO actions can improve the job performance? (Choose two.)

Accepted Answer

Use pushdown predicates to filter data at the source.. Option B is correct because pushdown predicates let AWS Glue push filtering logic down to the source RDS for MySQL database, so only the required incremental rows are read over JDBC instead of the entire table, reducing I/O and shuffle work in the job. Option D is correct because increasing the number of DPUs adds more Apache Spark executors and parallel task slots, which improves throughput for a large, daily incremental ETL workload that is currently resource-bound. Option A is not appropriate because switching to a Standard single-node worker removes distributed processing and would generally slow the job rather than improve performance. Option C is not appropriate because adding more transformations increases CPU and memory work in the ETL script, which would make the job slower, not faster. Option E is not appropriate because disabling output compression increases the volume of data written to Amazon S3 and read downstream, raising I/O and cost rather than improving job performance.

Answer

Change the worker type to Standard (single node).

Answer

Add more transformations to the ETL script to clean data.

Answer

Disable compression on the output data to reduce CPU usage.

Question 7

A media company is building a data pipeline to ingest user activity logs from multiple sources into Amazon S3. The logs are JSON files generated every minute. The company wants to use Amazon Athena to query the logs with minimal latency and cost. The current approach is to use AWS Kinesis Data Firehose to deliver the logs to S3 with a prefix like 'logs/2024/01/01/00/file.json'. However, when running Athena queries, the team notices high query costs because Athena scans all files in the 'logs/' prefix even when querying for a specific date. What should the team do to reduce the amount of data scanned by Athena?

Accepted Answer

Create a Hive-style partition structure in S3 with keys like 'year=2024/month=01/day=01/hour=00/' and update the Glue Data Catalog accordingly.. Athena reduces data scanned by using partition pruning, which requires a Hive-style partition structure in S3 (e.g., year=2024/month=01/day=01/hour=00/) registered in the Glue Data Catalog. When queries filter on partition columns, Athena only reads the relevant partitions instead of scanning the entire logs/ prefix. This directly addresses the high query cost caused by scanning all files.

Answer

Create an Athena view that filters by date.

Answer

Increase the number of partitions by using a more granular prefix like 'logs/2024/01/01/00/00/'.

Answer

Convert the JSON files to Apache Parquet format using AWS Glue ETL jobs.

Question 8

A company is using AWS Glue to run ETL jobs that transform data from Amazon DynamoDB to Amazon S3. The DynamoDB table has a large number of items (over 10 million) and is heavily used by production applications. The Glue job reads the entire DynamoDB table each time it runs, causing increased read capacity consumption and affecting production performance. The team wants to reduce the impact on the source DynamoDB table while still keeping the S3 data up-to-date. What should the team do?

Accepted Answer

Use DynamoDB Streams and AWS Lambda to capture changes and write them to S3, then run incremental Glue jobs.. Using DynamoDB Streams and AWS Lambda to capture changes and write them to S3, then running incremental Glue jobs, reduces the read load on the DynamoDB table because only changed data is processed. This approach keeps S3 data up-to-date without scanning the entire table, minimizing impact on production performance.

Answer

Increase the DynamoDB read capacity units to handle the Glue job's read load.

Answer

Use the DynamoDB console to export the table to S3 in Parquet format.

Answer

Reduce the parallelism of the Glue job to lower the read throughput.

Question 9

A data engineer is building an AWS Glue ETL job that reads a large Amazon S3 dataset of nested JSON files and must flatten the nested arrays into separate rows for downstream analytics. The engineer needs the most efficient, code-free way to apply this transformation within the Glue job. Which approach should the engineer use?

Accepted Answer

Use the AWS Glue Studio visual editor and add a Flatten transformation to the job.. The Flatten transformation in AWS Glue Studio is purpose-built to un-nest arrays and structs into rows without custom code, directly satisfying the need for an efficient, code-free flattening step in a Glue ETL job on nested JSON.

Answer

Configure the Glue crawler to automatically flatten nested JSON during cataloging.

Answer

Write a custom Python shell script in Glue to recursively flatten the JSON.

Answer

Use an Apache Spark filter transformation to explode the nested arrays.

Question 10

A data engineer runs an AWS Glue job that writes Parquet files to Amazon S3. The job frequently fails with an error indicating too many small files are being written, causing slow downstream Athena queries. The engineer wants to reduce the number of output files without changing the transformation logic. Which action should the engineer take?

Accepted Answer

Set the Glue job's output to use coalesce or repartition before writing.. Controlling output partitions with coalesce or repartition before writing is the direct way to reduce the number of small Parquet files in an AWS Glue job, improving downstream Athena performance without altering transformation logic.

Answer

Enable bookmarks on the Glue job to track processed files.

Answer

Change the output format from Parquet to CSV to reduce file count.

Answer

Increase the number of DPUs allocated to the Glue job.

Question 11

A data engineer needs to transform data in an AWS Glue job using a custom Python library that is not available by default. The library is packaged as a .whl file stored in Amazon S3. The engineer wants the Glue job to use this library without modifying the job script to install it at runtime. What should the engineer do?

Accepted Answer

Place the .whl file in the Glue job's Python library path parameter (--extra-py-files).. Specifying the wheel via the --extra-py-files job parameter makes the custom Python library available to the AWS Glue job without script changes, which is the correct way to include additional dependencies stored in Amazon S3.

Answer

Add the library to the Glue Data Catalog as a custom classifier.

Answer

Use a Glue development endpoint to install the library, then run the job there.

Answer

Upload the .whl to the Glue job's script bucket and import it directly by filename.

Question 12

A data engineer must ingest data from an Amazon DynamoDB table into an Amazon S3 bucket for analytics. The table receives continuous writes, and the ingestion must capture changes with minimal latency. The engineer wants a fully managed solution that requires no server provisioning. Which approach should the engineer use?

Accepted Answer

Enable DynamoDB Streams on the table, and use AWS Lambda to read the stream and write the data to Amazon S3.. DynamoDB Streams provides a time-ordered sequence of item-level changes, and Lambda can process these events in near real time. Writing to S3 from Lambda is a common serverless pattern that requires no server management and scales automatically. This combination meets the latency and managed-service requirements without the overhead of provisioning or full-table scans.

Answer

Use AWS Database Migration Service (AWS DMS) with ongoing replication to replicate the DynamoDB table to Amazon S3.

Answer

Use Amazon Kinesis Data Firehose to read directly from DynamoDB and deliver to Amazon S3.

Answer

Schedule an AWS Glue job to run every minute, reading the entire DynamoDB table and writing to Amazon S3.

Question 13

A data engineer is using AWS Glue to transform data from Amazon S3 and load it into Amazon Redshift. The job runs daily and processes 500 GB of data. The engineer notices that the job takes several hours and wants to optimize performance. The data is stored in Parquet format and partitioned by date. Which optimization should the engineer implement to improve the job's performance?

Accepted Answer

Use predicate pushdown to filter data at the source and partition pruning to read only necessary partitions.. Predicate pushdown and partition pruning are key optimizations for AWS Glue jobs reading partitioned Parquet data. They minimize the amount of data scanned by pushing filters to the source and skipping irrelevant partitions. This reduces I/O and compute time, directly addressing the performance issue without unnecessary cost increases.

Answer

Convert the Parquet data to CSV to improve read performance.

Answer

Increase the number of DPUs for the Glue job to scale horizontally.

Answer

Enable job bookmarks to track processed data and avoid reprocessing.

Question 14

A data engineer must transform data in Amazon S3 using Apache Spark. The transformation logic needs to be reused across multiple AWS Glue jobs, and the engineer wants to version-control the code and run it in a serverless environment without managing clusters. Which approach should the engineer take?

Accepted Answer

Package the transformation logic as a Python wheel file stored in Amazon S3, reference it in AWS Glue job parameters, and import it as a module in the job script.. AWS Glue supports referencing external Python libraries through the --extra-py-files job parameter. Packaging transformation logic as a wheel file in Amazon S3 allows the code to be versioned, reused across multiple Glue jobs, and executed in a serverless Spark environment. This satisfies both the reusability and serverless requirements without duplicating code or introducing cluster management overhead.

Answer

Create an AWS Lambda function containing the transformation logic and invoke it from each AWS Glue job using boto3.

Answer

Provision an Amazon EMR cluster, store the transformation code in a Git repository, and run Spark jobs on the cluster.

Answer

Create an AWS Glue job with a script that contains all transformation logic, and copy-paste the code into each job that needs it.

Question 15

A data engineer is using AWS Database Migration Service (AWS DMS) to replicate ongoing changes from an on-premises Oracle database to an Amazon S3 bucket in near real time. The source table has a primary key and the database is configured for ARCHIVELOG mode. The engineer needs change data capture (CDC) to capture INSERT, UPDATE, and DELETE operations. Which AWS DMS task setting should be used to capture ongoing changes?

Accepted Answer

Set the migration type to 'full-load-and-cdc' and enable supplemental logging on the source table.. To capture ongoing changes from Oracle to S3 with AWS DMS, the task must be configured with migration type 'full-load-and-cdc' and supplemental logging must be enabled on the source. Supplemental logging provides the additional column data needed for UPDATE and DELETE operations. Without it, CDC cannot function correctly. The other options either use incorrect migration types or omit the critical supplemental logging requirement.

Answer

Set the migration type to 'full-load' and configure ongoing replication using Oracle GoldenGate.

Answer

Set the migration type to 'cdc' only and rely on the Oracle redo logs without supplemental logging.

Answer

Set the migration type to 'full-load-and-cdc' and increase the commit rate to reduce latency.