DEA-C01 Data Ingestion and Transformation Practice Test 9 — 15 Questions

Question 1

A data engineer is designing a data pipeline that ingests CSV files from an FTP server to Amazon S3. The files arrive hourly and each file is about 500 MB. The engineer wants to minimize operational overhead and cost. Which approach is best?

Accepted Answer

Use AWS Transfer for SFTP and point the endpoint to an S3 bucket. AWS Transfer for SFTP provides a fully managed FTP service that writes directly to S3, eliminating the need to manage servers. Lambda with boto3 is code-heavy; EC2 requires management; Snowball is for large offline transfers.

Answer

Write a Python script in AWS Lambda using boto3 to download from FTP and upload to S3

Answer

Use AWS Snowball Edge to transfer files weekly

Answer

Deploy an Amazon EC2 instance with a cron job to run wget and aws s3 cp

Question 2

A company needs to transform JSON data from Amazon Kinesis Data Streams into Parquet format and store it in Amazon S3. The transformation includes simple field mappings and type conversions. Which approach is most cost-effective and serverless?

Accepted Answer

Use Amazon Kinesis Data Firehose with an AWS Lambda function for transformation and output to Parquet. Kinesis Data Firehose with built-in Lambda transformation can convert JSON to Parquet efficiently. Glue is heavier and more expensive for simple transforms; EC2 is not serverless; SageMaker is ML-focused.

Answer

Use Amazon EC2 instances running Apache Spark Streaming

Answer

Use Amazon SageMaker for transformation

Answer

Use an AWS Glue ETL job triggered by a Kinesis stream

Question 3

A data engineer needs to ingest data from an on-premises Apache Kafka cluster into Amazon S3. The data volume is about 10 TB per day. The engineer wants to set up a managed Kafka connector. Which AWS service should they use?

Accepted Answer

Amazon MSK Connect. Amazon MSK Connect is a fully managed Kafka Connect service that can run S3 sink connectors. Lambda is not a Kafka sink; Kinesis is a different streaming service; DMS is for databases.

Answer

AWS Database Migration Service

Answer

AWS Lambda with Kafka trigger

Answer

Amazon Kinesis Data Streams

Question 4

A company is ingesting CSV files into Amazon S3. Each file contains a header row. The pipeline uses AWS Glue to crawl the S3 bucket and create a table in the AWS Glue Data Catalog. However, the crawler is including the header as data. What is the most likely cause?

Accepted Answer

The crawler is configured to use a custom classifier that does not skip headers. By default, Glue crawlers do not skip headers unless configured to do so. Adding an explicit 'skip.header.line.count=1' in the table properties fixes it. The crawler classifies correctly but needs the property.

Answer

The S3 bucket has versioning enabled

Answer

The crawler is running in 'crawl all folders' mode

Answer

The CSV files are not compressed

Question 5

A data engineer needs to ingest data from an Amazon S3 bucket into an Amazon Redshift table on a daily schedule. The data is in CSV format and the schema matches. Which service is simplest for this batch ingestion?

Accepted Answer

Amazon Redshift COPY command. Redshift COPY command loads from S3 efficiently. Glue and Data Pipeline also work but are more complex. Athena is for querying, not loading.

Answer

AWS Glue ETL job with JDBC connection

Answer

AWS Data Pipeline

Answer

Amazon Athena CREATE TABLE AS SELECT

Question 6

A company is ingesting streaming data from a fleet of weather sensors. Each sensor sends a JSON payload every second. The data is used for real-time dashboarding and also archived to S3. The pipeline should handle sudden bursts of data without data loss. Which architecture meets these requirements?

Accepted Answer

Amazon Kinesis Data Streams -> AWS Lambda for dashboard -> Amazon Kinesis Data Firehose -> S3. Kinesis Data Streams provides durable storage and can handle bursts; Firehose reads from the stream and writes to S3; Lambda can update dashboards. Direct Firehose ingestion may throttle; SQS is not ideal for streaming; EC2 is not serverless.

Answer

Amazon EC2 with Apache Kafka -> S3

Answer

Amazon Kinesis Data Firehose directly with no buffer

Answer

Amazon SQS -> AWS Lambda -> S3

Question 7

A data engineer is ingesting XML data from an external API into Amazon S3. The engineer needs to transform the XML to JSON using AWS Glue. The XML structure is deeply nested. Which Apache Spark method should be used in the Glue ETL script?

Accepted Answer

Use spark.read.format('xml') with Databricks XML library. Spark's Databricks XML library (spark-xml) can parse XML and convert to DataFrame; then write as JSON. The 'com.databricks.spark.xml' format is the standard. AWS Glue's built-in 'xml' option is limited; 'from_xml' is not a Spark method; 'XmlInputFormat' is Hadoop MR.

Answer

Use the built-in AWS Glue 'xml' data source

Answer

Use Hadoop's XmlInputFormat

Answer

Use the Spark SQL function from_xml()

Question 8

A company is ingesting large volumes of sensor data into Amazon S3. The data must be encrypted at rest using an AWS KMS customer managed key. Which TWO actions are required to enable server-side encryption with AWS KMS (SSE-KMS) on the S3 bucket?

Accepted Answer

Set the default encryption on the S3 bucket to AWS-KMS and specify the KMS key. Option B is correct because setting the default encryption on the S3 bucket to AWS-KMS and specifying the KMS key ensures that all objects uploaded to the bucket are automatically encrypted with SSE-KMS using that customer managed key. This is the primary configuration step to enforce server-side encryption at rest with a customer managed key.

Answer

Enable S3 Versioning on the bucket

Answer

Enable Amazon CloudWatch Logs for the bucket

Answer

Add a bucket policy that denies uploads without encryption

Question 9

A data engineer is designing a data ingestion pipeline for real-time clickstream data. The data must be available for both real-time analytics and batch processing. The engineer wants to use Amazon Kinesis Data Streams. Which THREE components should be included in the architecture?

Accepted Answer

Amazon Kinesis Data Streams. Kinesis Data Streams ingests data, Kinesis Client Library (KCL) consumes for real-time analytics, and Kinesis Data Firehose delivers to S3 for batch. Kinesis Data Analytics is for real-time SQL, not batch; SQS is not used; Glue is batch but not directly from stream.

Answer

Amazon Kinesis Data Analytics

Answer

Amazon SQS as a buffer

Question 10

A company is ingesting Apache logs from multiple web servers into AWS. The logs are sent via Amazon CloudWatch Logs to a subscription filter that delivers to a Lambda function. The Lambda function parses the logs and writes to Amazon S3. However, there is a significant backlog. Which THREE actions can reduce the backlog?

Accepted Answer

Increase the Lambda function memory allocation. Increasing Lambda concurrency, increasing memory (which also increases CPU), and using a reserved concurrency ensure capacity. Changing runtime has minor effect; SQS is not used in this path.

Answer

Route the CloudWatch Logs subscription to an Amazon SQS queue first

Answer

Change the Lambda function runtime from Python to Node.js

Question 11

Refer to the exhibit. An IAM policy for an AWS Lambda function. The Lambda function is triggered by an S3 event (object created) and needs to read from a Kinesis stream. However, the function fails with access denied when trying to read from Kinesis. What is the most likely cause?

Accepted Answer

The Kinesis stream is encrypted with a customer managed KMS key, and the Lambda function lacks kms:Decrypt permission. The Lambda function lacks permission to list the Kinesis stream (kinesis:ListStreams). The policy only allows DescribeStream, GetShardIterator, and GetRecords. To read from a stream, the initial DescribeStream call works, but the Lambda execution role may need additional actions depending on the SDK. However, often the error is due to missing kinesis:ListStreams if the SDK lists streams first. But more commonly, the policy is correct; the failure might be due to missing s3:ListBucket? No, the error is Kinesis. Actually, the policy allows GetRecords, so the most likely cause is that the Kinesis stream is encrypted with a KMS key and the Lambda function lacks kms:Decrypt permission. This is a common scenario. The policy does not include KMS permissions.

Answer

The Lambda function is not in the same region as the Kinesis stream

Answer

The Lambda function does not have permission to list S3 buckets

Answer

The S3 bucket policy denies access to the Lambda function

Question 12

Refer to the exhibit. A CloudFormation stack outputs the Glue job name and S3 bucket names. The Glue job transforms CSV files from the raw bucket to Parquet in the processed bucket. However, the Glue job is failing with an error that it cannot write to the processed bucket. What is the most likely cause?

Accepted Answer

The Glue job does not have permission to write to the processed bucket. The Glue job's IAM role likely does not have s3:PutObject permission on the processed bucket. The CloudFormation outputs show the bucket names, but the role may only have read access to raw. The error is write-related, not read.

Answer

The raw data bucket is in a different region

Answer

The Glue job is not using the correct worker type

Answer

The Glue job is using an incorrect file format

Question 13

Refer to the exhibit. A Lambda function named 'IngestionProcessor' is failing. The engineer checks CloudWatch Logs and sees the log group exists but storedBytes is 0. Why might the logs show no data?

Accepted Answer

The Lambda execution role does not have permission to write logs to CloudWatch. The log group exists but has no log streams, meaning the function hasn't executed (or its execution role lacks logs:CreateLogStream and logs:PutLogEvents). The most common cause is missing IAM permissions for CloudWatch Logs.

Answer

The Lambda function is configured with a dead letter queue

Answer

The Lambda function has not been invoked yet

Answer

The log group is encrypted with a KMS key and the Lambda function lacks decrypt permission

Question 14

A company needs to ingest real-time clickstream data from a web application into Amazon S3 for analytics. The data must be available within minutes of generation. Which AWS service should be used to capture and deliver this streaming data?

Accepted Answer

Amazon Kinesis Data Firehose. Option A (Amazon Kinesis Data Firehose) is correct because it can capture streaming data and deliver it to S3 with minimal latency (typically 60 seconds). Option B (AWS Glue) is a batch ETL service, not real-time. Option C (Amazon SQS) is a message queue, not designed for direct S3 delivery. Option D (Amazon RDS) is a relational database.

Answer

Amazon RDS

Answer

AWS Glue

Answer

Amazon Simple Queue Service (SQS)

Question 15

A data engineer is tasked with transforming JSON data from an S3 bucket into Parquet format for efficient querying. The transformation should run on a schedule every hour. Which AWS service is best suited for this task?

Accepted Answer

AWS Glue. Option C (AWS Glue) is correct because it offers scheduled ETL jobs that can read from S3, transform data, and write back in Parquet format. Option A (Amazon Athena) is a query service, not for scheduled transformations. Option B (Amazon EMR) is more complex and typically used for large-scale processing. Option D (AWS Lambda) has a 15-minute timeout and is not ideal for hourly batch jobs.

Answer

AWS Lambda

Answer

Amazon Athena

Answer

Amazon EMR