DEA-C01 Data Ingestion and Transformation Practice Test 36 — 15 Questions

Question 1

Refer to the exhibit. A data engineer has attached this IAM policy to an AWS Glue job role. The Glue job fails when trying to write transformed data to an S3 bucket located in a different AWS account. What is the most likely reason?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The policy does not grant s3:ListBucket, and the bucket policy may not allow cross-account access. The IAM policy shown does not include the s3:ListBucket permission, which is required for the Glue job to list objects in the S3 bucket before writing. Additionally, cross-account access requires both the source account's IAM policy (this one) to grant write permissions and the target account's S3 bucket policy to explicitly allow the source account's role, which may not be configured. Without s3:ListBucket, the Glue job cannot verify the bucket's existence or structure, causing the write operation to fail.

Answer

The policy does not allow lambda:InvokeAsync

Answer

The Glue job role does not have permissions to write to S3

Answer

The policy does not include kinesis:DescribeStream

Question 2

A data engineer needs to load data from an on-premises Oracle database to Amazon S3 daily. The table is 500 GB and grows by 50 MB per day. The load must capture only new and changed rows since the last run. Which solution is MOST cost-effective and requires the least maintenance?

Accepted Answer

Use AWS Database Migration Service (DMS) with ongoing replication (CDC). AWS DMS with ongoing replication (CDC) is the most cost-effective and low-maintenance solution because it continuously captures only new and changed rows from the Oracle source using its built-in CDC mechanism (reading redo logs), without requiring custom scripting or full table exports. It automatically handles schema changes, resumability, and incremental loading to S3, minimizing operational overhead and data transfer costs.

Answer

Write a custom Python script on EC2 to query the Oracle redo logs and upload to S3

Answer

Export the entire table to CSV daily using a script and upload to S3

Answer

Use AWS Glue ETL job with a JDBC connection and a timestamp filter

Question 3

A company uses AWS Glue to process JSON logs from S3. The logs have a nested structure and the schema evolves over time. The data engineer needs to ensure the Glue job can handle schema changes without failing. Which configuration should be used?

Accepted Answer

Set the job parameter '--enable-glue-datacatalog' and '--mergeDynamicColumns' to true. Setting '--enable-glue-datacatalog' allows the Glue job to use the Data Catalog as the metastore, and '--mergeDynamicColumns' (or the equivalent '--enable-schema-evolution' in newer Glue versions) instructs the job to dynamically merge new columns from the evolving JSON schema into the existing table schema during runtime, preventing job failures due to schema mismatches. This is specifically designed for nested, schema-evolving data like JSON logs, as it automatically reconciles differences between the source data and the catalog definition.

Answer

Manually update the table schema in the Glue Data Catalog before each run

Answer

Use Spark SQL with a static schema definition in the script

Answer

Enable AWS Glue Schema Registry and define a schema version

Question 4

A company needs to ingest data from multiple SaaS applications (Salesforce, Marketo) into Amazon S3 for analytics. The data volume is moderate (~100 GB per day). The pipeline must handle schema changes, deduplicate records, and provide low latency (under 1 hour). Which THREE services should be used? (Choose THREE.)

Accepted Answer

Amazon AppFlow. Amazon AppFlow is the correct choice because it is a fully managed integration service specifically designed to transfer data from SaaS applications like Salesforce and Marketo to AWS services such as Amazon S3. It supports incremental transfers, handles schema changes automatically via its schema evolution feature, and can achieve sub-hour latency for moderate data volumes (~100 GB/day) without custom coding.

Answer

Amazon Kinesis Data Streams

Answer

AWS Database Migration Service (DMS)

Question 5

A company needs to ingest data from a MySQL database into Amazon S3 in near real-time. The database is running on EC2. The data engineer wants to minimize the impact on the source database. Which service should be used?

Accepted Answer

AWS Database Migration Service (DMS) with ongoing replication. AWS DMS with ongoing replication (change data capture) is the correct choice because it can continuously replicate changes from a MySQL source database to Amazon S3 with minimal performance impact. DMS uses a transactional log-based approach (MySQL binlog) to capture changes as they occur, avoiding heavy SELECT queries on the source. This enables near real-time ingestion without adding significant load to the production database.

Answer

AWS Glue ETL job with a JDBC connection

Answer

Amazon RDS for MySQL with read replica

Answer

AWS Schema Conversion Tool (SCT)

Question 6

A data engineer uses AWS Glue to process data from S3. The Glue job frequently fails with 'Out of Memory' errors. The job reads several large compressed files. What is the MOST effective way to resolve this issue without changing the code?

Accepted Answer

Increase the number of G.1X workers or use G.2X workers. Increasing the number of G.1X workers or switching to G.2X workers directly addresses the 'Out of Memory' errors by allocating more memory per Spark executor. G.1X provides 16 GB of memory per worker, while G.2X provides 32 GB, which is critical when processing large compressed files because decompression and transformation require additional heap space. This approach resolves the issue without modifying the job code, as it only changes the resource configuration.

Answer

Convert the compressed files to uncompressed format before processing

Answer

Repartition the data to fewer partitions

Answer

Increase the job timeout setting

Question 7

A company is building a data lake on S3. They have a large volume of CSV files (hundreds of GB) in a source bucket. They need to convert them to Parquet, partition by date, and ensure the data is encrypted at rest with SSE-KMS. The pipeline must be triggered automatically when new files arrive. Which THREE steps should be part of the solution? (Choose THREE.)

Accepted Answer

Configure S3 Event Notification to send events to an SQS queue. S3 Event Notifications can be configured to send events to an SQS queue when new CSV files arrive. This decouples the ingestion pipeline, allowing the Glue job to poll the queue for new file notifications and trigger processing without tight coupling or polling the S3 bucket directly. SQS provides reliable, scalable message delivery that can trigger downstream ETL workflows.

Answer

Use Amazon Kinesis Data Firehose to ingest new files

Answer

Use Amazon Athena CTAS query to convert files in batch

Question 8

A data engineer needs to ingest data from an on-premises Oracle database to Amazon S3 daily. The data volume is 500 GB per day, and the network bandwidth is 200 Mbps. The requirement is to minimize the impact on the source database and ensure data integrity. Which combination of AWS services should be used?

Accepted Answer

AWS Database Migration Service (DMS) with S3 as target. AWS DMS with S3 as target is correct because it supports continuous change data capture (CDC) from Oracle, minimizing impact on the source database by reading redo logs instead of querying tables directly. It can handle 500 GB/day over 200 Mbps (which yields ~2.16 TB/day theoretical max) and ensures data integrity via transactional consistency and validation checksums. DMS also automatically partitions large datasets and can resume from failures, making it ideal for daily bulk loads.

Answer

AWS Glue ETL jobs with JDBC connection

Answer

Amazon Kinesis Data Firehose with Oracle as source

Answer

AWS Data Pipeline with SQLActivity

Question 9

A data engineering team needs to transform CSV files to Parquet format after they land in an S3 bucket. The transformation should be triggered automatically as soon as a new file arrives. Which AWS service is best suited for this task?

Accepted Answer

AWS Lambda function triggered by S3 event. AWS Lambda functions can be directly triggered by S3 events (e.g., `s3:ObjectCreated:*`) to process newly uploaded CSV files. This serverless approach provides near-instantaneous, event-driven transformation to Parquet without managing any infrastructure, making it the most cost-effective and simplest solution for this specific use case.

Answer

AWS Batch job submitted by S3 event

Answer

Amazon EMR cluster running continuously

Answer

AWS Glue ETL job scheduled every 5 minutes

Question 10

A company uses AWS Glue ETL jobs to process data from an S3 data lake. The job reads data in CSV format, transforms it, and writes to Parquet. The job runs daily and takes 2 hours to complete. The data volume is increasing by 20% each month. The engineer wants to reduce the job runtime. Which action is most effective?

Accepted Answer

Partition the input data in S3 by date and use partition pruning in the job. Most effective because partitioning the input data by date and using partition pruning allows the Glue ETL job to read only the relevant partitions instead of scanning the entire S3 data lake. This drastically reduces the amount of data processed, which directly addresses the growing data volume and shortens job runtime. Partition pruning is a core optimization for Spark-based Glue jobs, as it leverages Hive-style partitioning to skip unnecessary files.

Answer

Increase the number of DPUs for the Glue job

Answer

Enable compression on the input CSV files

Answer

Switch from Python Shell to Spark ETL

Question 11

A company uses Amazon Kinesis Data Streams to ingest IoT sensor data. The data is processed by an AWS Lambda function that transforms the records and writes to an Amazon S3 bucket. Recently, the Lambda function has been failing with 'Rate exceeded' errors for the S3 PUT API calls. The data volume is 10 MB/s with average record size 2 KB. What should be done to resolve this issue?

Accepted Answer

Add a random prefix to the S3 object key to distribute writes across multiple prefixes. The 'Rate exceeded' error for S3 PUT API calls indicates that the Lambda function is hitting S3 request rate limits. S3 buckets have a default limit of 3,500 PUT requests per second per prefix. With a data volume of 10 MB/s and an average record size of 2 KB, the Lambda function is generating approximately 5,000 PUT requests per second (10 MB/s ÷ 2 KB), which exceeds the per-prefix limit. Adding a random prefix to the S3 object key distributes writes across multiple prefixes, effectively increasing the aggregate request rate limit to 3,500 PUT requests per second per prefix, thereby resolving the throttling issue.

Answer

Switch to Amazon Kinesis Data Firehose to write to S3

Answer

Increase the Lambda function's reserved concurrency

Answer

Increase the number of Kinesis shards

Question 12

A company uses AWS Glue DataBrew for data preparation. The data source is an S3 bucket with millions of small CSV files (each < 1 MB). The DataBrew project takes a long time to load the sample data. What is the most likely cause and solution?

Accepted Answer

The large number of small files causes S3 LIST overhead; concatenate files into larger files. DataBrew loads a sample of the data by listing objects in the S3 bucket. With millions of small CSV files, the S3 LIST API call becomes a bottleneck because each list operation has a 1000-object limit per response, requiring multiple paginated requests. Concatenating the small files into larger files reduces the number of objects, dramatically decreasing LIST overhead and speeding up sample loading.

Answer

Use Amazon Athena to query the data instead of DataBrew

Answer

The DataBrew job is under-provisioned; increase the number of DPUs

Answer

Use AWS Glue ETL instead of DataBrew for this volume

Question 13

A company is streaming IoT data from thousands of devices into Amazon Kinesis Data Streams. The data must be transformed in real time before being stored in Amazon S3. Which service should be used to perform the transformation as the data streams through Kinesis?

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Analytics for Apache Flink is the correct choice because it is purpose-built for running Apache Flink applications that can perform real-time transformations, filtering, and enrichment on data streaming through Kinesis Data Streams before outputting the results to destinations like Amazon S3. It integrates natively with Kinesis Data Streams as a source and can write transformed data directly to S3 using a Flink sink, making it ideal for this streaming ETL use case.

Answer

AWS Glue

Answer

Amazon EMR

Answer

AWS Lambda

Question 14

A data engineer is designing a data ingestion pipeline for clickstream data from a mobile app. The data volume varies, with occasional spikes up to 10 MB/s. The pipeline must persist the raw data in Amazon S3 and make it available for near-real-time analytics via Amazon Athena. Which combination of services minimizes cost and operational overhead?

Accepted Answer

Amazon Kinesis Data Firehose with direct delivery to Amazon S3, then Amazon Athena. Amazon Kinesis Data Firehose is the most cost-effective and low-overhead solution for ingesting variable-volume clickstream data (up to 10 MB/s) into Amazon S3 because it is a fully managed service that automatically scales, buffers, and compresses data before delivery. It integrates directly with S3 without requiring custom code or infrastructure management, and the data is immediately queryable by Amazon Athena with no additional transformation steps.

Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics, then Amazon S3

Answer

Amazon SQS with an Auto Scaling group of EC2 instances writing to Amazon S3

Answer

Amazon Kinesis Data Streams with AWS Lambda for transformation, then Amazon S3

Question 15

A company is using Amazon Kinesis Data Firehose to deliver streaming data to Amazon S3. The data must be transformed from JSON to Parquet format before landing in S3. The transformation logic is simple: convert the JSON schema to Parquet. Which approach meets the requirements with the least operational overhead?

Accepted Answer

Use the built-in data format conversion feature of Firehose with an AWS Glue Data Catalog table. Amazon Kinesis Data Firehose provides a built-in data format conversion feature that can automatically convert incoming JSON data to Parquet format using an AWS Glue Data Catalog table as the schema reference. This approach requires no custom code, no additional infrastructure, and no manual transformation logic, making it the simplest solution with the least operational overhead for a straightforward JSON-to-Parquet conversion.

Answer

Use an AWS Lambda function to transform records to Parquet before sending to Firehose

Answer

Use Amazon Kinesis Data Analytics to convert the stream to Parquet

Answer

Provision an Amazon EMR cluster to convert the data in micro-batches