DEA-C01 Data Operations and Support — 25 Questions

Question 1

A data engineer notices that an AWS Glue job processing data from an Amazon S3 bucket frequently fails with 'OutOfMemoryError'. The job reads CSV files, applies transformations, and writes Parquet to another S3 bucket. The job has 10 workers of type G.1X. Which change is MOST likely to resolve the issue?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Change the worker type from G.1X to G.2X. The G.1X worker type provides 16 GB of memory per worker. An OutOfMemoryError indicates that the job's memory requirements exceed this limit. Upgrading to G.2X doubles the memory per worker to 32 GB, directly addressing the memory shortage without changing the parallelism or incurring the overhead of additional workers.

Answer

Increase the number of workers to 20

Answer

Change the worker type from G.1X to G.8X

Answer

Enable the Spark UI to monitor memory and tune the job

Question 2

A company uses Amazon DynamoDB as a source for an AWS Glue job. The job reads a large table using a DynamoDB export to S3 feature. The job is failing with 'ThrottlingException' from DynamoDB. What should the data engineer do to resolve this issue WITHOUT changing the job's logic?

Accepted Answer

Use the DynamoDB export to S3 feature and read the exported data from S3. The DynamoDB export to S3 feature creates a point-in-time snapshot of the table data in S3 without consuming any read capacity units (RCUs) from the DynamoDB table. By reading the exported data from S3 instead of directly scanning the DynamoDB table, the Glue job avoids triggering ThrottlingException entirely, as the export operation uses the table's backup and restore mechanism, not the read path. This resolves the issue without altering the job's logic, as the job can be reconfigured to read from the S3 export location.

Answer

Use DynamoDB Streams to capture changes and process them incrementally

Answer

Reduce the number of DynamoDB read segments in the Glue job

Answer

Increase the read capacity units (RCU) of the DynamoDB table

Question 3

A data engineer is monitoring an Amazon Kinesis Data Analytics application that uses a SQL query to aggregate streaming data. The application is falling behind and the millisBehindLatest metric is increasing. Which action should the engineer take to improve performance?

Accepted Answer

Increase the Parallelism setting of the Kinesis Data Analytics application. Increasing the Parallelism setting of the Kinesis Data Analytics application allows the SQL query to process data across more in-application streams and operators concurrently, directly addressing the lag indicated by the rising millisBehindLatest metric. This action scales the compute resources allocated to the application without changing the source stream or the query logic, making it the most direct way to improve throughput for a SQL-based Kinesis Data Analytics application.

Answer

Switch from SQL to Apache Flink for the analytics application

Answer

Increase the number of shards in the source Kinesis stream

Answer

Decrease the window duration of the SQL query

Question 4

A data engineer is troubleshooting an AWS Glue job that reads from an Amazon RDS for PostgreSQL database using a JDBC connection. The job fails with the error 'java.sql.SQLException: No suitable driver'. Which TWO actions should the engineer take to resolve this issue? (Select TWO.)

Accepted Answer

Verify that the connection string in the job's JDBC URL uses the correct format and includes the driver class. The error 'java.sql.SQLException: No suitable driver' specifically indicates that the JDBC driver class cannot be loaded or matched, so option A is correct: the engineer must verify the JDBC URL format (e.g., jdbc:postgresql://host:5432/dbname) and ensure it references the correct driver class so the DriverManager can locate a registered driver. Option D is also correct because AWS Glue does not bundle the PostgreSQL JDBC driver by default; the engineer must upload the postgresql-<version>.jar to Amazon S3 and add it as a dependent JAR (via --extra-jars or the job's dependent libraries) so the driver class is on the classpath. Option B is not the cause here because VPC and security group issues produce connection timeouts or 'connection refused' errors, not 'No suitable driver'. Option C is irrelevant since a timeout value does not affect driver class loading. Option E is incorrect because IAM permissions govern AWS API authorization, not JDBC driver resolution, and 'rds:*' would not fix a missing driver.

Answer

Check that the Glue job's VPC and security groups allow outbound traffic to the RDS instance

Answer

Restart the Glue job with a higher timeout value

Answer

Update the IAM role associated with the Glue job to allow 'rds:*' permissions

Question 5

An AWS Glue job that processes streaming data from Amazon Kinesis Data Streams is failing intermittently with 'Failed to checkpoint' errors. The job uses checkpointing to an Amazon S3 bucket every 60 seconds. Which action should the engineer take to resolve the issue?

Accepted Answer

Increase the checkpoint interval to 120 seconds.. The 'Failed to checkpoint' error in AWS Glue streaming jobs typically occurs when the checkpoint operation exceeds the 60-second interval due to high throughput or large state size. Increasing the checkpoint interval to 120 seconds provides more time for the checkpoint to complete, reducing the likelihood of timeouts and allowing the job to stabilize without losing progress.

Answer

Move the checkpoint location to an Amazon DynamoDB table.

Answer

Decrease the Kinesis shard count to reduce throughput.

Answer

Disable checkpointing and rely on Kinesis iterator age.

Question 6

A company runs a batch ETL job on Amazon EMR every night. Recently, the job started failing with 'Out of Memory' errors in the Spark executors. The data volume has grown 20% in the past month. The cluster uses uniform instance groups with 5 core nodes of r5.xlarge (4 vCPU, 32 GB RAM). Which change should the data engineer implement to resolve the issue with minimal cost increase?

Accepted Answer

Configure instance fleets to include r5.xlarge and r5.2xlarge instances.. Using instance fleets allows the cluster to include both r5.xlarge and r5.2xlarge instances, enabling the Spark executors to use the larger instances for memory-intensive tasks while still leveraging the existing r5.xlarge nodes. This provides a cost-effective way to handle the 20% data growth by adding memory capacity without replacing the entire cluster or over-provisioning all nodes. Instance fleets also support Spot Instances, which can further reduce costs while addressing the Out of Memory errors.

Answer

Increase the number of core nodes to 7.

Answer

Change instance type to r5.2xlarge (8 vCPU, 64 GB RAM) for all nodes.

Answer

Tune Spark memory configurations to reduce executor memory overhead.

Question 7

A data engineer is troubleshooting a nightly AWS Glue ETL job that reads from an Amazon RDS for MySQL table and writes to an Amazon S3 bucket in Parquet format. The job runs successfully most days, but occasionally fails with the error 'ERROR: An error occurred while calling o67.pyWriteDynamicFrame. The transaction log for the database is full due to 'LOG_BACKUP'.' What is the MOST likely cause of this error?

Accepted Answer

The MySQL database transaction log needs to be backed up to free space.. The error message 'The transaction log for the database is full due to 'LOG_BACKUP'' indicates that the MySQL database's transaction log has reached its maximum size because it has not been backed up and truncated. In MySQL, the transaction log (often the InnoDB redo log or binary log) must be backed up periodically to free space; otherwise, write operations fail. This is a database-side issue, not a Glue or permissions problem, so the correct action is to back up the transaction log to release space.

Answer

The MySQL database has reached its maximum number of concurrent connections.

Answer

The AWS Glue job does not have sufficient permissions to write to the S3 bucket.

Answer

The AWS Glue job is configured with an incorrect 'writeDynamicFrame' method.

Question 8

A company runs a data pipeline that ingests clickstream data from a web application into Amazon Kinesis Data Streams. A Lambda function processes records from the stream and writes them to an Amazon S3 bucket in JSON format. The pipeline has been running smoothly, but for the past hour, the Lambda function has been failing with 'Rate exceeded' errors, and the Kinesis stream shows elevated 'IteratorAgeMilliseconds' metrics. The Lambda function has a reserved concurrency of 100, and the Kinesis stream has 10 shards. The average record size is 5 KB, and the data rate is approximately 15 MB per second. Which combination of actions should a data engineer take to resolve the issue and prevent recurrence? (Choose TWO.)

Accepted Answer

Increase the Lambda function's reserved concurrency to 200.. Option A is correct because the 'Rate exceeded' errors indicate the Lambda function is hitting its concurrency ceiling: with 10 shards, Kinesis invokes one Lambda invocation per shard concurrently, but a reserved concurrency of 100 can be throttled when the function's per-shard processing spawns additional invocations or when other functions compete for the account's concurrency pool, so raising reserved concurrency to 200 gives the function more headroom to process records without throttling. Option B is correct because the stream's data rate of 15 MB/s exceeds the ingest capacity of 10 shards: each Kinesis shard supports 1 MB/s ingress (and 2 MB/s egress), so 10 shards provide only 10 MB/s, causing the elevated IteratorAgeMilliseconds; increasing to 20 shards provides 20 MB/s of ingest capacity, allowing the Lambda function to keep up and reduce iterator age. Option C is not correct because reducing batch size would lower throughput per invocation and could worsen iterator age rather than resolve the concurrency throttling. Option D is not correct because S3 multipart upload addresses large object upload performance, not Lambda concurrency or Kinesis shard capacity. Option E is not correct because replacing Lambda with Kinesis Data Firehose would change the pipeline architecture and is not a targeted fix for the concurrency and shard-capacity issues described.

Answer

Decrease the Lambda function's batch size from 100 to 50.

Answer

Enable S3 multipart upload for the Lambda function.

Answer

Replace the Lambda function with Amazon Kinesis Data Firehose to write directly to S3.

Question 9

A company uses AWS Glue to run ETL jobs that process data from Amazon S3 and write results to Amazon Redshift. The Glue job uses the JDBC connection to Redshift. Recently, the job has been failing intermittently with the error: 'java.sql.SQLException: [Amazon](500310) Invalid operation: INSERT has more expressions than target columns;' The Glue job writes to a staging table in Redshift before performing a merge into the final table. The staging table schema matches the source data. The error occurs only on some days and affects different columns each time. The data engineer suspects that the source data occasionally contains extra columns due to a schema drift in the upstream data producer. Which approach should the data engineer take to handle this issue robustly?

Accepted Answer

Use a Glue DynamicFrame and apply the resolveChoice method to make the schema consistent.. Glue DynamicFrames can automatically handle schema drift using the `resolveChoice` method, which allows you to specify how to handle columns that appear inconsistently across records (e.g., making them null, casting to a common type, or dropping them). This directly addresses the intermittent error caused by extra columns in the source data without requiring manual schema updates or fragile conditional logic.

Answer

Skip any records that have extra columns by adding a conditional check in the Glue script.

Answer

Manually update the Redshift staging table schema whenever the source data changes.

Answer

Use a Glue DynamicFrame and apply the dropFields method to remove extra columns before writing.

Question 10

A data engineer is troubleshooting a Glue ETL job that reads from an S3 bucket and writes to a Redshift table. The job fails with a 'MemoryError' when processing a large dataset. Which TWO actions should the engineer take to resolve this issue? (Choose TWO.)

Accepted Answer

Increase the number of DPUs and set 'spark.sql.shuffle.partitions' to a higher value.. Increasing the number of DPUs (Data Processing Units) provides more memory and compute resources to the Glue job, directly addressing the MemoryError. Setting 'spark.sql.shuffle.partitions' to a higher value reduces the amount of data shuffled per partition, preventing out-of-memory errors during wide transformations like joins or aggregations.

Answer

Increase the number of DPUs and set 'coalesce(1)' in the script.

Answer

Decrease the number of DPUs and increase 'spark.shuffle.partitions'.

Answer

Set the 'RedshiftTempDir' parameter to a larger S3 bucket.

Question 11

A data engineer applies the above S3 bucket policy to an S3 bucket used by a Glue ETL job. The Glue job writes objects to the bucket. Which of the following is true about the behavior of the policy?

Accepted Answer

The policy denies all PutObject requests because the Allow and Deny statements are contradictory.. In AWS IAM policy evaluation, an explicit Deny always overrides any Allow. The policy has an Allow statement granting s3:PutObject for all principals, but a separate Deny statement explicitly denies s3:PutObject when the encryption condition is not aws:kms. Since the Deny applies to all PutObject requests (including those with no encryption or AES256), and the Allow does not include a condition to match only aws:kms, the Deny takes precedence and blocks all PutObject requests, making the policy effectively deny all PutObject operations.

Answer

The policy allows PutObject with aws:kms encryption because the Allow statement is broader.

Answer

The policy allows PutObject with no encryption because the Deny only applies to PutObject.

Answer

The policy allows PutObject with AES256 encryption and denies PutObject with aws:kms encryption.

Question 12

A company runs a nightly batch processing pipeline using AWS Glue ETL jobs. The pipeline reads data from an Amazon S3 bucket, transforms it, and writes results to an Amazon Redshift cluster. Recently, the data volume has increased significantly, and some Glue jobs are failing with the error 'java.lang.OutOfMemoryError: Java heap space'. The data engineer needs to modify the job configuration to prevent these failures without changing the code. The job currently uses 10 DPUs and processes data in a single Spark DataFrame. Which of the following is the MOST effective solution?

Accepted Answer

Increase the number of DPUs to 20 and enable job bookmarking for incremental processing.. Increasing DPUs from 10 to 20 provides more memory and compute resources, directly addressing the 'java.lang.OutOfMemoryError: Java heap space' caused by insufficient memory for the single DataFrame. Enabling job bookmarking allows incremental processing, which reduces the volume of data processed per run, further mitigating memory pressure without code changes.

Answer

Reduce the number of DPUs to 5 and increase the Spark executor memory by setting 'spark.executor.memory' in job parameters.

Answer

Change the script to use DynamicFrame instead of DataFrame and disable the 'spark.sql.shuffle.partitions' configuration.

Answer

Add a 'coalesce(1)' operation before writing to Redshift to reduce the number of output files.

Question 13

Match each AWS data compression format to its typical use case.

Accepted Answer

Snappy: Designed for high speed and efficient compression, ideal for real-time data processing.. In AWS data engineering, compression format choice depends on trade-offs between compression ratio and speed. Snappy and LZO are optimized for speed, Gzip and Bzip2 for higher compression (with Bzip2 highest), and Zstandard offers a middle ground. Common misassociations include confusing Gzip with Bzip2 for highest compression and missing Snappy's speed focus.

Answer

Gzip: Provides the highest compression ratio, best for archival storage.

Answer

Bzip2: Commonly used for log files due to good compression ratio.

Question 14

A company runs a data pipeline using AWS Lambda to process records from an Amazon Kinesis Data Stream. Recently, the Lambda function has been experiencing high invocation errors and the stream is throttling. The function performs simple transformations and writes to Amazon S3. What is the most effective way to reduce throttling and errors?

Accepted Answer

Increase the batch size in the Lambda event source mapping.. Increasing the batch size in the Lambda event source mapping allows each invocation to process more records from the Kinesis stream, reducing the number of total invocations. This lowers the rate at which Lambda polls the stream, which decreases the likelihood of hitting the Kinesis read throughput limits (5 transactions per second per shard) and reduces throttling errors. The simple transformations and S3 writes are likely I/O-bound, so larger batches improve throughput without increasing invocation concurrency.

Answer

Increase the Lambda function timeout.

Answer

Enable provisioned concurrency on the Lambda function.

Answer

Increase the number of shards in the Kinesis stream.

Question 15

A company runs a critical data pipeline using Amazon Kinesis Data Firehose to deliver data to Amazon S3. The data is compressed with GZIP and partitioned by year/month/day/hour. Recently, the delivery to S3 has been failing with 'Rate exceeded' errors. The Firehose delivery stream has a buffer size of 128 MB and buffer interval of 60 seconds. What is the most effective way to resolve this issue?

Accepted Answer

Increase the buffer size to 256 MB and buffer interval to 120 seconds.. The 'Rate exceeded' error indicates that Kinesis Data Firehose is sending requests to S3 at a rate that exceeds the S3 bucket's request rate limits for PUT operations. Increasing the buffer size to 256 MB and the buffer interval to 120 seconds allows Firehose to accumulate more data before each S3 PUT request, reducing the number of requests per second and staying within S3's 3,500 PUT requests per second limit per prefix. This directly addresses the throttling issue without changing the data volume.

Answer

Transition objects to S3 Glacier after 30 days.

Answer

Decrease the buffer size to 64 MB and buffer interval to 30 seconds.

Answer

Enable server-side encryption on the S3 bucket.

Question 16

A data engineer is monitoring an Amazon EMR cluster running a Spark job. The job is processing a large dataset and the engineer notices that the cluster is using a high percentage of disk space on the core nodes. The job fails with 'No space left on device' error. What is the most effective way to resolve this issue without modifying the job logic?

Accepted Answer

Increase the number of core nodes in the cluster.. Increasing the number of core nodes distributes the intermediate shuffle data and temporary files across more nodes, reducing the per-node disk usage. This directly addresses the 'No space left on device' error without altering the Spark job logic, as core nodes in EMR store both HDFS data and local shuffle spills.

Answer

Attach additional EBS volumes to the core nodes.

Answer

Increase the EBS volume size attached to the core nodes.

Answer

Change the core node instance type to one with more memory.

Question 17

A company uses Amazon CloudWatch Logs to collect application logs from EC2 instances. The logs are exported to Amazon S3 for long-term storage. Recently, the export task failed with the error 'Access Denied'. What is the most likely cause of this failure?

Accepted Answer

The IAM role does not have s3:PutObject permission on the destination bucket.. The export task from CloudWatch Logs to S3 uses an IAM role to write data to the destination bucket. If the role lacks the s3:PutObject permission, the S3 service will reject the request with an 'Access Denied' error. This is the most common cause because the export operation requires write access to the bucket.

Answer

The S3 bucket policy denies access from the CloudWatch Logs service.

Answer

The IAM role does not have s3:ListBucket permission.

Answer

The EC2 instances are in a VPC without a VPC endpoint for CloudWatch Logs.

Question 18

A data engineer is troubleshooting an AWS Glue crawler that is not correctly inferring the schema of CSV files stored in Amazon S3. The files have headers, but the crawler is treating the header row as data. The crawler is configured with a custom classifier that has a CSV classifier with 'Column header' set to 'Use first row as header'. What is the most likely reason the crawler is not recognizing the header?

Accepted Answer

The CSV classifier's 'Quote symbol' setting does not match the files.. The custom CSV classifier's 'Quote symbol' setting must match the quoting character used in the CSV files. If the files use double quotes (the default) but the classifier is set to a different quote symbol (e.g., single quote), the classifier will misinterpret the first row as data rather than recognizing it as a header. This is because the classifier uses the quote symbol to parse fields correctly; a mismatch causes the header row to be treated as a regular data row.

Answer

The CSV files have a varying number of columns across rows.

Answer

The CSV files have a different delimiter than the default comma.

Answer

The header row contains uppercase letters.

Question 19

A company is using AWS Glue ETL jobs to process data from Amazon S3 and write results back to S3. The jobs are failing intermittently with 'ThrottlingException' errors. Which TWO configurations would help reduce these errors?

Accepted Answer

Add retry logic with exponential backoff in the job script.. Option C is correct because AWS Glue API calls that return ThrottlingException are transient, and implementing retry logic with exponential backoff in the job script allows the job to gracefully re-attempt throttled requests after progressively longer waits, reducing intermittent failures. Option E is correct because increasing the number of DPUs scales the job's compute capacity, which can reduce the frequency of throttling by allowing the workload to complete faster and by distributing the request rate more efficiently across the allocated resources. Options A and D do not belong because decreasing DPUs reduces capacity and would likely worsen throttling, and switching from Spark to Python shell changes the execution environment rather than addressing the throttling root cause. Option B does not belong because GZIP compression reduces output data size and storage/transfer costs but does not directly mitigate ThrottlingException errors from AWS service API rate limits.

Answer

Decrease the number of DPUs for the job.

Answer

Enable GZIP compression on the output data.

Answer

Change the job type from Spark to Python shell.

Question 20

A company ingests IoT sensor data into an S3 bucket. Daily, a Lambda function reads new objects, processes them, and writes results to a DynamoDB table. Recently, the Lambda function started timing out after 15 minutes. The data volume has increased, and the function processes records one by one. Which solution would improve performance without significant cost increase?

Accepted Answer

Use S3 Batch Operations to invoke the Lambda function in parallel for each object.. S3 Batch Operations invokes the Lambda function for each object in parallel, efficiently handling increased volume without significant cost increase. Option A is incorrect because AWS Glue ETL jobs have startup overhead and may cost more. Option B is incorrect because increasing the timeout does not address the root cause of sequential processing; the function would still process records one by one and may still timeout. Option D is incorrect because increasing DynamoDB write capacity does not speed up the Lambda processing; the bottleneck is the sequential processing within the function.

Answer

Replace Lambda with an AWS Glue ETL job.

Answer

Increase the Lambda function timeout to 30 minutes.

Answer

Increase the DynamoDB write capacity units.