DEA-C01 Hard Questions — 25 Difficult Questions

Question 1

A data engineering team uses Amazon Kinesis Data Analytics for Apache Flink to process streaming data. They notice that the application's checkpointing is failing intermittently, causing data reprocessing. The application uses a large state. Which configuration change should the team make to improve checkpoint reliability?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Increase the checkpointing interval.. Increasing the checkpointing interval reduces the frequency of checkpoint operations, giving the system more time to complete each checkpoint before the next one starts. This alleviates backpressure and resource contention, which is critical when dealing with large state, as checkpointing large state is I/O and CPU intensive and can fail if intervals are too tight.

Answer

Disable checkpointing to avoid failures.

Answer

Switch the state backend from in-memory to RocksDB.

Answer

Increase the parallelism of the application.

Question 2

A company is using Amazon DynamoDB with on-demand capacity for a gaming application that experiences unpredictable traffic spikes. The application consistently sees 'ProvisionedThroughputExceededException' errors during spikes. The data engineer needs to resolve this issue without changing the application code. What should the engineer do?

Accepted Answer

Switch the table to on-demand capacity mode. The application is already using on-demand capacity, but the error 'ProvisionedThroughputExceededException' indicates the table is actually in provisioned mode, not on-demand. Switching to on-demand capacity mode eliminates throttling by automatically scaling throughput to match traffic spikes, with no code changes required.

Answer

Enable DynamoDB Accelerator (DAX) to cache read requests

Answer

Increase the read capacity units

Answer

Enable auto scaling for the table with a higher maximum capacity

Question 3

A company is using Amazon RDS for SQL Server with Multi-AZ. The database has a 500 GB data file and 100 GB log file. The application experiences high latency during peak hours. Monitoring shows high WriteIOPS on the primary. Which change will reduce latency without losing the ability to failover?

Accepted Answer

Increase the provisioned IOPS on the RDS instance. The high WriteIOPS on the primary indicates that the storage layer is saturated, causing latency. Increasing provisioned IOPS on the RDS instance directly addresses the bottleneck by providing more I/O capacity, and since Multi-AZ is already enabled, the standby remains synchronized and failover capability is preserved.

Answer

Reduce the log file size by changing recovery model

Answer

Create a Read Replica in a different Availability Zone

Answer

Switch to Multi-AZ with two readable standbys

Question 4

A data engineer is designing a streaming pipeline that ingests data from an Amazon Kinesis Data Stream (with 5 shards) into Amazon S3. The data must be transformed using a complex stateful operation that cannot be done in a Lambda function (limited to 15 minutes). The engineer needs a solution that can maintain state across multiple records. Which service should be used?

Accepted Answer

Amazon Kinesis Data Analytics for Apache Flink. Amazon Kinesis Data Analytics for Apache Flink is the correct choice because it supports stateful stream processing with exactly-once semantics, can maintain state across multiple records, and has no 15-minute execution limit like AWS Lambda. It natively integrates with Kinesis Data Streams and can sink transformed data to S3, meeting all requirements for complex stateful operations.

Answer

Amazon EMR running Spark Structured Streaming

Answer

Amazon Kinesis Data Firehose with Lambda transformation

Answer

AWS Glue streaming ETL job

Question 5

A company is using AWS Glue to run ETL jobs that write data to an Amazon S3 data lake. The jobs are failing with '503 Slow Down' errors. The data engineering team has already implemented retries. What is the BEST long-term solution?

Accepted Answer

Implement a backoff strategy to reduce the request rate.. The '503 Slow Down' error from Amazon S3 indicates that the request rate is too high and S3 is throttling the requests. The best long-term solution is to implement a backoff strategy (exponential backoff) to reduce the request rate, which allows the Glue job to automatically slow down and retry with increasing delays, aligning with S3's request rate limits and avoiding sustained throttling.

Answer

Enable S3 Transfer Acceleration.

Answer

Use S3 multipart upload for all objects.

Answer

Increase the number of retries in the Glue job.

Question 6

Which THREE factors should a data engineer consider when choosing between Amazon RDS and Amazon DynamoDB for a new application? (Choose THREE.)

Accepted Answer

Requirement for ACID transactions across multiple tables.. Amazon RDS (with engines like PostgreSQL or MySQL) supports ACID transactions across multiple tables, ensuring atomicity, consistency, isolation, and durability for complex operations. DynamoDB, while supporting transactions, is limited to a single AWS account and region, and its transactional API has constraints on item sizes and throughput, making RDS the better choice for multi-table ACID compliance.

Answer

Ability to encrypt data at rest.

Answer

Support for multi-region disaster recovery.

Question 7

A data engineer is designing a data ingestion pipeline for a social media analytics platform. The pipeline must ingest tweets in real-time, perform sentiment analysis, and store results in Amazon S3. The sentiment analysis is compute-intensive and must be done as the data arrives. The estimated throughput is 10,000 tweets per second. Which architecture is most suitable?

Accepted Answer

Amazon Kinesis Data Streams with Amazon Kinesis Data Analytics for sentiment analysis, then Kinesis Data Firehose to S3.. The most suitable because Amazon Kinesis Data Streams can ingest up to 10,000 records per second per shard (with shard-level scaling), and Kinesis Data Analytics provides built-in, low-latency stream processing for compute-intensive sentiment analysis using SQL or Apache Flink. Kinesis Data Firehose then reliably buffers and writes the processed results to Amazon S3 without custom code, ensuring near-real-time delivery.

Answer

Amazon SQS with AWS Lambda pollers to process tweets and store in S3.

Answer

Amazon EMR with Spark Streaming to process tweets and write to S3.

Answer

Amazon API Gateway with AWS Lambda to process each tweet and store in S3.

Question 8

A healthcare company processes patient records in near-real-time using Amazon Kinesis Data Streams. Each record contains sensitive personal health information (PHI). The data must be encrypted at rest and in transit. The company also needs to audit access to the data. The data engineer is designing the ingestion pipeline. Which combination of services and configurations meets these requirements?

Accepted Answer

Use Kinesis Data Streams with SSE-KMS and TLS, and enable CloudTrail for data events.. Kinesis Data Streams supports server-side encryption (SSE) using AWS KMS for at-rest encryption, and TLS for in-transit. CloudTrail can log Kinesis API calls for auditing. Option A lacks encryption at rest. Option B lacks auditing. Option D is wrong because S3 does not replace Kinesis for streaming.

Answer

Use Kinesis Data Firehose to deliver data to S3 with SSE-S3, and enable CloudTrail for S3.

Answer

Use Kinesis Data Streams with TLS and enable CloudTrail for auditing. Do not enable SSE.

Answer

Use Kinesis Data Streams with SSE-KMS and TLS. Do not enable any auditing.

Question 9

A company runs a data pipeline that uses Amazon EMR to process large datasets. The pipeline reads data from S3, processes it using Spark, and writes results back to S3. Recently, the pipeline has been failing with 'OutOfMemoryError' in the Spark executors. The EMR cluster is configured with 5 core nodes of type m5.xlarge (4 vCPU, 16 GB memory each). The Spark application uses dynamic allocation and default Spark configurations. The input data size is approximately 500 GB in Parquet format. What is the most cost-effective way to resolve the out-of-memory errors?

Accepted Answer

Change the core node instance type to r5.xlarge (32 GB memory) and keep 5 nodes.. The current cluster has limited memory per node (16 GB). By switching to memory-optimized instances like r5.xlarge (32 GB), each node has double the memory, reducing the chance of OOM. This is more cost-effective than adding more nodes because the total memory per node increases without increasing the number of instances. Option A is wrong because increasing the number of nodes adds more memory but also more cost; it might be more expensive than using fewer, larger nodes. Option C is wrong because it's generally not recommended to increase spark.executor.memory beyond the physical memory; it could cause YARN to kill containers. Option D is wrong because Parquet is already efficient; changing to a different format may not solve memory issues.

Answer

Increase the spark.executor.memory setting to 8 GB in the Spark configuration.

Answer

Increase the number of core nodes to 10 to distribute the data across more executors.

Answer

Change the input data format from Parquet to ORC to reduce memory footprint.

Question 10

A CloudFormation template includes this IAM policy for a cross-account S3 upload use case. What is the purpose of the condition?

Accepted Answer

To ensure that uploaded objects grant full control to the bucket owner.. The condition in the IAM policy uses the `s3:x-amz-acl` key with a value of `bucket-owner-full-control`. This ensures that any object uploaded to the S3 bucket explicitly grants the bucket owner full control over the object, overriding the default behavior where the uploading account retains ownership. This is critical in cross-account uploads to prevent the uploading account from retaining exclusive access to the objects.

Answer

To enforce server-side encryption with KMS.

Answer

To limit the size of objects that can be uploaded.

Answer

To restrict uploads to only a specific AWS account.

Question 11

A data engineer is troubleshooting a Kinesis Data Firehose delivery stream that is experiencing high error rates when writing to an S3 bucket. The error logs indicate 'AccessDenied' errors. The S3 bucket policy allows access from the Firehose service, but the errors persist. What is the most likely cause?

Accepted Answer

The IAM role assumed by Firehose does not have the s3:PutObject permission. The most likely cause is that the IAM role assumed by Kinesis Data Firehose lacks the `s3:PutObject` permission. Even if the S3 bucket policy allows access from the Firehose service, the IAM role must explicitly grant the necessary S3 write permissions for Firehose to deliver data. Without this permission, Firehose receives 'AccessDenied' errors when attempting to write objects to the bucket.

Answer

The S3 bucket has a lifecycle policy that is deleting objects too quickly

Answer

The S3 bucket has default encryption enabled

Answer

The S3 bucket uses an AWS KMS key for encryption and Firehose does not have kms:Decrypt permission

Question 12

A data engineer is designing a data lake on Amazon S3 for analytics. The data includes sensitive PII that must be encrypted at rest. The company requires that the encryption keys be managed by the company's own hardware security module (HSM) and rotated every 90 days. Which TWO options meet these requirements? (Choose TWO.)

Accepted Answer

Use S3 server-side encryption with customer-provided keys (SSE-C). SSE-C allows you to provide your own encryption keys, which can be managed and rotated from your own HSM. The keys are used server-side by S3 to encrypt objects at rest, but S3 does not store the keys—you manage them entirely, meeting the requirement for key management on your own HSM with 90-day rotation.

Answer

Use S3 server-side encryption with AWS KMS and an AWS managed key

Answer

Use client-side encryption with keys stored in AWS Secrets Manager

Answer

Use S3 server-side encryption with S3 managed keys (SSE-S3)

Question 13

Which THREE factors should a data engineer consider when choosing between Amazon S3 and Amazon DynamoDB for storing time-series data? (Choose three.)

Accepted Answer

Required query complexity (simple key lookups vs. range scans). Amazon S3 supports range scans via its ListObjectsV2 API with prefix and delimiter parameters, but it is not optimized for complex queries like filtering on non-key attributes or aggregations. DynamoDB, on the other hand, excels at simple key lookups and range scans on its sort key, but lacks native support for complex query patterns such as multi-attribute filtering or joins. Therefore, the required query complexity directly influences the choice: S3 is better for simple prefix-based scans, while DynamoDB is better for key-value lookups and sort-key range queries.

Answer

Cost per GB of storage

Answer

Total data volume

Question 14

A company runs a nightly AWS Glue ETL job that writes results to an Amazon Redshift table using the JDBC connector. Recently, the job has been failing with the error 'ERROR:  connection to server at ... failed: server closed the connection unexpectedly'. The Redshift cluster is in a private subnet with a VPC endpoint for S3. The Glue job runs in the same VPC with enhanced VPC routing enabled. Which is the most likely cause?

Accepted Answer

SSL is not enabled on the Redshift cluster.. The error 'server closed the connection unexpectedly' typically indicates that the Redshift cluster rejected the connection, often because SSL is required but not enabled in the Glue JDBC connection. Redshift clusters can be configured with the `require_ssl` parameter set to true, forcing all connections to use SSL. Option B is correct because without SSL, the server terminates the connection. Option A is incorrect because the 'redshift' compatibility mode is not a standard requirement; the JDBC driver works without it. Option C is incorrect because security group issues would cause a timeout or 'connection refused', not a server-side close. Option D is incorrect because Glue supports Redshift as a data source.

Answer

The JDBC driver is missing the 'redshift' compatibility mode setting.

Answer

The Glue job's security group does not allow outbound traffic to the Redshift cluster.

Answer

AWS Glue does not support Redshift as a data source.

Question 15

A company has an Amazon RDS for MySQL database that is experiencing performance issues due to a large number of read requests. The application is read-heavy and can tolerate eventually consistent reads. Which action will reduce the load on the primary database with the least operational overhead?

Accepted Answer

Create a read replica in the same region. Creating a read replica in the same region offloads read traffic from the primary RDS instance to a read-only copy, which directly addresses the read-heavy workload. Since the application can tolerate eventually consistent reads, the slight replication lag is acceptable, and this solution requires minimal operational overhead—just a few clicks in the AWS console or a single API call—without any application code changes.

Answer

Use Amazon ElastiCache for caching

Answer

Enable Multi-AZ deployment

Answer

Increase the instance size of the primary DB

Question 16

A company uses Amazon DynamoDB to store session data for a web application. The application experiences occasional spikes in traffic, causing throttling on the table. The data engineer needs to implement a solution that handles traffic spikes without manual intervention and minimizes cost. What should the data engineer do?

Accepted Answer

Enable DynamoDB Auto Scaling for the table.. DynamoDB Auto Scaling (option D) automatically adjusts the provisioned read and write capacity based on actual traffic patterns, using CloudWatch alarms and the Application Auto Scaling service. This handles traffic spikes without manual intervention and minimizes cost by scaling down during low traffic periods, making it the ideal solution for variable workloads like session data.

Answer

Switch to provisioned capacity with a high fixed read/write capacity.

Answer

Implement DynamoDB Accelerator (DAX) to cache read requests.

Answer

Purchase DynamoDB reserved capacity.

Question 17

A data engineering team is managing an Amazon Redshift cluster that is used for BI reporting. The cluster has a mix of large tables (some over 1 TB) and many smaller tables. The team notices that queries on a large fact table are slow. The fact table is distributed using KEY distribution on the customer_id column, which has high cardinality. The team wants to improve query performance. They have the option to change the distribution style and sort key. Which redesign should they implement?

Accepted Answer

Change the distribution style to EVEN and set the sort key to a date column used in WHERE clauses.. Using EVEN distribution ensures data is evenly distributed across all nodes, avoiding data skew that can occur with KEY distribution on a high-cardinality column like customer_id. Setting the sort key to a date column used in WHERE clauses enables range-restricted scans, significantly reducing the amount of data scanned for common BI queries that filter by date. This combination improves query performance by maximizing parallelism and minimizing I/O.

Answer

Keep the distribution style as AUTO and set the sort key to customer_id.

Answer

Change the distribution style to ALL and set the sort key to customer_id.

Answer

Change the distribution style to KEY on a different column with high cardinality.

Question 18

Which THREE of the following are benefits of using Amazon DynamoDB Accelerator (DAX)? (Choose three.)

Accepted Answer

Offloads read traffic from the DynamoDB table.. DAX acts as a read-through cache that offloads read traffic from the DynamoDB table, reducing the number of read requests that hit the underlying table and thus lowering the consumed read capacity units (RCUs). This allows the table to handle more concurrent reads without scaling up provisioned capacity.

Answer

Improves write throughput by batching writes.

Answer

Supports write-through caching to improve write performance.

Question 19

Refer to the exhibit. A data engineer has attached this bucket policy to an S3 bucket named data-lake-bucket. The engineer wants to allow only GET requests from the corporate network (10.0.0.0/16) over HTTPS. However, users report that they cannot access objects even when connected to the corporate network. What is the issue?

Accepted Answer

The Deny statement blocks all requests that are not using HTTPS, including those from the corporate network.. The Deny statement with `aws:SecureTransport` set to `false` blocks all HTTP requests. Since the Allow statement only permits GET requests from the corporate network (10.0.0.0/16) but does not require HTTPS, any request from that network that uses HTTP is denied by the explicit Deny. The Deny statement overrides the Allow, so even legitimate corporate users are blocked if they use HTTP.

Answer

The Deny statement should include a condition on the source IP.

Answer

The Allow statement should include a condition for SecureTransport.

Answer

The Allow statement should specify s3:GetObject instead of s3:GetObject.

Question 20

A data engineering team is ingesting data from multiple sources into Amazon S3 using AWS Glue ETL jobs. The jobs are failing intermittently with the error: 'Task ran out of memory'. The input data size varies widely from 100 MB to 10 GB per job. Which configuration change would best mitigate this issue?

Accepted Answer

Enable job bookmarking to process only incremental data. Enabling job bookmarking allows the Glue ETL job to process only incremental (new or changed) data, which directly addresses the intermittent out-of-memory errors caused by widely varying input sizes (100 MB to 10 GB). By skipping previously processed data, the job consistently handles smaller data volumes per run, reducing memory pressure on the Spark executors.

Answer

Increase the number of workers in the Glue job

Answer

Reduce the batch size in the S3 source node

Answer

Change the job type from Spark to Python shell