DEA-C01 Data Operations and Support — Timed 100 Questions

Question 1

A data engineer notices that an AWS Glue job processing data from an Amazon S3 bucket frequently fails with 'OutOfMemoryError'. The job reads CSV files, applies transformations, and writes Parquet to another S3 bucket. The job has 10 workers of type G.1X. Which change is MOST likely to resolve the issue?

Accepted Answer

Change the worker type from G.1X to G.2X. The G.1X worker type provides 16 GB of memory per worker. An OutOfMemoryError indicates that the job's memory requirements exceed this limit. Upgrading to G.2X doubles the memory per worker to 32 GB, directly addressing the memory shortage without changing the parallelism or incurring the overhead of additional workers.

Answer

Increase the number of workers to 20

Answer

Change the worker type from G.1X to G.8X

Answer

Enable the Spark UI to monitor memory and tune the job

Question 2

A company uses Amazon Kinesis Data Streams to ingest clickstream data. The data is consumed by a custom consumer application that writes to Amazon S3 every 5 minutes. The consumer is falling behind and processing lag is increasing. Which action is MOST effective to reduce the lag?

Accepted Answer

Increase the number of shards in the Kinesis stream. The consumer is falling behind because the stream's throughput capacity is insufficient for the incoming data volume. Increasing the number of shards in the Kinesis stream directly increases the total read capacity (each shard provides 2 MB/s read throughput and 5 transactions/second), allowing the consumer to process more data in parallel and reduce lag.

Answer

Switch to Amazon Kinesis Data Firehose to deliver data directly to S3

Answer

Increase the batch size of records written to S3

Answer

Reduce the retention period of the stream

Question 3

A data team runs a daily AWS Glue ETL job that processes data from an Amazon Redshift cluster and writes results to Amazon S3. The job completes successfully but takes 2 hours longer than expected. The job uses the JDBC connection to Redshift. The Redshift cluster is 4 dc2.large nodes. The Glue job has 10 workers of type G.1X. Which change would MOST likely reduce the job duration?

Accepted Answer

Use the S3 staging option in the Glue connection to unload data from Redshift to S3 first. The JDBC connection in AWS Glue reads data row-by-row from Redshift, which is slow for large datasets. By enabling the S3 staging option in the Glue connection, the job uses Redshift's UNLOAD command to export data to S3 in parallel, then Glue reads from S3. This bypasses the JDBC bottleneck and leverages Redshift's massively parallel processing (MPP) to export data much faster.

Answer

Use Redshift Spectrum to query data directly from S3

Answer

Increase the Redshift cluster size to 8 nodes

Answer

Increase the number of Glue workers to 20

Question 4

A company uses Amazon DynamoDB as a source for an AWS Glue job. The job reads a large table using a DynamoDB export to S3 feature. The job is failing with 'ThrottlingException' from DynamoDB. What should the data engineer do to resolve this issue WITHOUT changing the job's logic?

Accepted Answer

Use the DynamoDB export to S3 feature and read the exported data from S3. Option C is correct because the DynamoDB export to S3 feature creates a point-in-time snapshot of the table data in S3 without consuming any read capacity units (RCUs) from the DynamoDB table. By reading the exported data from S3 instead of directly scanning the DynamoDB table, the Glue job avoids triggering ThrottlingException entirely, as the export operation uses the table's backup and restore mechanism, not the read path. This resolves the issue without altering the job's logic, as the job can be reconfigured to read from the S3 export location.

Answer

Use DynamoDB Streams to capture changes and process them incrementally

Answer

Reduce the number of DynamoDB read segments in the Glue job

Answer

Increase the read capacity units (RCU) of the DynamoDB table

Question 5

A data engineer is monitoring an Amazon Kinesis Data Analytics application that uses a SQL query to aggregate streaming data. The application is falling behind and the millisBehindLatest metric is increasing. Which action should the engineer take to improve performance?

Accepted Answer

Increase the Parallelism setting of the Kinesis Data Analytics application. Increasing the Parallelism setting of the Kinesis Data Analytics application allows the SQL query to process data across more in-application streams and operators concurrently, directly addressing the lag indicated by the rising millisBehindLatest metric. This action scales the compute resources allocated to the application without changing the source stream or the query logic, making it the most direct way to improve throughput for a SQL-based Kinesis Data Analytics application.

Answer

Switch from SQL to Apache Flink for the analytics application

Answer

Increase the number of shards in the source Kinesis stream

Answer

Decrease the window duration of the SQL query

Question 6

A data engineer is troubleshooting an AWS Glue job that reads from an Amazon RDS for PostgreSQL database using a JDBC connection. The job fails with the error 'java.sql.SQLException: No suitable driver'. Which TWO actions should the engineer take to resolve this issue? (Select TWO.)

Accepted Answer

Verify that the connection string in the job's JDBC URL uses the correct format and includes the driver class. Option A is correct because the 'No suitable driver' error in JDBC indicates that the driver class specified in the JDBC URL is either missing or incorrect. For PostgreSQL, the JDBC URL must follow the format 'jdbc:postgresql://host:port/database' and the driver class must be 'org.postgresql.Driver'. If the URL is malformed or the driver class is not properly referenced, the Glue job cannot load the driver, leading to this specific SQLException.

Answer

Check that the Glue job's VPC and security groups allow outbound traffic to the RDS instance

Answer

Restart the Glue job with a higher timeout value

Answer

Update the IAM role associated with the Glue job to allow 'rds:*' permissions

Question 7

A company uses Amazon S3 to store raw data and runs AWS Glue ETL jobs to transform it into Parquet. The data is then queried using Amazon Athena. Queries are slow and expensive due to high scan volumes. Which THREE design changes can improve query performance and reduce costs? (Select THREE.)

Accepted Answer

Convert the data to a columnar format like Parquet or ORC if not already. Option B is correct because columnar formats like Parquet or ORC store data by column rather than by row, allowing Athena to read only the columns needed for a query. This drastically reduces the amount of data scanned per query, directly lowering both latency and cost since Athena charges based on the volume of data read.

Answer

Increase the number of files by reducing file size to 1 MB

Answer

Use bucketing on high-cardinality columns

Question 8

A data engineer runs a Spark job on Amazon EMR that reads data from Amazon S3 and writes results back to S3. The job fails with an 'S3AccessDenied' error. The engineer verifies that the IAM role attached to the EMR cluster has s3:GetObject and s3:PutObject permissions on the relevant buckets. What is the MOST likely cause of the error?

Accepted Answer

The IAM role does not have s3:ListBucket permission on the bucket.. The IAM role attached to the EMR cluster must have the s3:ListBucket permission on the bucket to allow the Spark job to enumerate objects when reading from S3. Without this permission, even with s3:GetObject and s3:PutObject, the job fails with an 'S3AccessDenied' error because the S3 list operation is required for directory listing and file discovery.

Answer

S3 Transfer Acceleration is not enabled on the bucket.

Answer

EMRFS consistent view is not configured.

Answer

The S3 bucket is in a different AWS Region than the EMR cluster.

Question 9

An AWS Glue job that processes streaming data from Amazon Kinesis Data Streams is failing intermittently with 'Failed to checkpoint' errors. The job uses checkpointing to an Amazon S3 bucket every 60 seconds. Which action should the engineer take to resolve the issue?

Accepted Answer

Increase the checkpoint interval to 120 seconds.. The 'Failed to checkpoint' error in AWS Glue streaming jobs typically occurs when the checkpoint operation exceeds the 60-second interval due to high throughput or large state size. Increasing the checkpoint interval to 120 seconds provides more time for the checkpoint to complete, reducing the likelihood of timeouts and allowing the job to stabilize without losing progress.

Answer

Move the checkpoint location to an Amazon DynamoDB table.

Answer

Decrease the Kinesis shard count to reduce throughput.

Answer

Disable checkpointing and rely on Kinesis iterator age.

Question 10

A company uses AWS DMS to migrate data from an on-premises Oracle database to Amazon Redshift. The migration is successful, but after a few days, data in Redshift becomes inconsistent with the source due to ongoing changes. The company needs to keep Redshift synchronized with minimal latency. Which approach should the data engineer use?

Accepted Answer

Configure DMS with ongoing replication using change data capture (CDC).. AWS DMS supports ongoing replication using change data capture (CDC), which captures incremental changes from the Oracle source (via Oracle LogMiner or binary logs) and applies them to Amazon Redshift in near real-time. This approach ensures that Redshift remains synchronized with the source database with minimal latency, meeting the requirement for ongoing consistency after the initial full load.

Answer

Use Amazon Redshift COPY with S3 staging and AWS Lambda triggers.

Answer

Schedule a full DMS load every night.

Answer

Set up Amazon Redshift Spectrum to query the Oracle database directly.

Question 11

A data engineer notices that an Amazon Kinesis Data Firehose delivery stream is failing to deliver data to an Amazon S3 bucket. The CloudWatch metrics show 'DeliveryToS3.Success' is 0 and 'S3.BucketExists' is 1. What is the MOST likely cause?

Accepted Answer

The IAM role for Firehose lacks s3:PutObject permission.. The metric 'S3.BucketExists' is 1, confirming the S3 bucket exists, so the issue is not bucket existence. With 'DeliveryToS3.Success' at 0, the failure is in the write operation. The IAM role assumed by Firehose must have the s3:PutObject permission to deliver data; lacking it would cause all delivery attempts to fail silently, matching the observed metrics.

Answer

The S3 bucket has an ACL that denies access to Firehose.

Answer

The Firehose delivery stream Lambda transformation function is failing.

Answer

The S3 bucket does not exist.

Question 12

A company runs a batch ETL job on Amazon EMR every night. Recently, the job started failing with 'Out of Memory' errors in the Spark executors. The data volume has grown 20% in the past month. The cluster uses uniform instance groups with 5 core nodes of r5.xlarge (4 vCPU, 32 GB RAM). Which change should the data engineer implement to resolve the issue with minimal cost increase?

Accepted Answer

Configure instance fleets to include r5.xlarge and r5.2xlarge instances.. Option C is correct because using instance fleets allows the cluster to include both r5.xlarge and r5.2xlarge instances, enabling the Spark executors to use the larger instances for memory-intensive tasks while still leveraging the existing r5.xlarge nodes. This provides a cost-effective way to handle the 20% data growth by adding memory capacity without replacing the entire cluster or over-provisioning all nodes. Instance fleets also support Spot Instances, which can further reduce costs while addressing the Out of Memory errors.

Answer

Increase the number of core nodes to 7.

Answer

Change instance type to r5.2xlarge (8 vCPU, 64 GB RAM) for all nodes.

Answer

Tune Spark memory configurations to reduce executor memory overhead.

Question 13

A data engineer is troubleshooting an AWS Glue ETL job that fails with the error: 'An error occurred while calling o123.pyWriteDynamicFrame. Access Denied when writing to S3 bucket: my-bucket'. The job uses a Glue service role named 'GlueServiceRole'. Which TWO actions should the engineer take to resolve the issue? (Choose TWO.)

Accepted Answer

Check if the S3 bucket policy denies access from the GlueServiceRole.. Option C is correct because the error message indicates an access denied when writing to S3, which can be caused by a bucket policy that explicitly denies the Glue service role's access, even if the IAM policy allows it. Option D is correct because the IAM policy attached to GlueServiceRole must include the s3:PutObject permission on the specific bucket to allow the Glue job to write data.

Answer

Disable S3 Block Public Access on the bucket.

Answer

Grant the GlueServiceRole permission to write to the AWS Glue Data Catalog.

Answer

Ensure the Glue job is in the same VPC as the S3 bucket.

Question 14

A data engineer is monitoring an Amazon Kinesis Data Analytics for Apache Flink application that processes streaming data. The application is falling behind (increasing 'MillisBehindLatest') and the CPU utilization of the Flink task managers is consistently above 80%. Which THREE actions should the engineer take to improve performance? (Choose THREE.)

Accepted Answer

Increase the number of shards in the Kinesis data stream.. Increasing the number of shards in the Kinesis data stream (Option A) directly increases the ingestion capacity and parallelism source for the Flink application. With more shards, the application can read data from more partitions concurrently, reducing the backlog indicated by 'MillisBehindLatest'. This is a fundamental scaling action for Kinesis-based Flink applications.

Answer

Decrease the checkpoint interval to reduce state size.

Answer

Decrease the number of task managers to reduce CPU contention.

Question 15

A data engineer is troubleshooting a nightly AWS Glue ETL job that reads from an Amazon RDS for MySQL table and writes to an Amazon S3 bucket in Parquet format. The job runs successfully most days, but occasionally fails with the error 'ERROR: An error occurred while calling o67.pyWriteDynamicFrame. The transaction log for the database is full due to 'LOG_BACKUP'.' What is the MOST likely cause of this error?

Accepted Answer

The MySQL database transaction log needs to be backed up to free space.. The error message 'The transaction log for the database is full due to 'LOG_BACKUP'' indicates that the MySQL database's transaction log has reached its maximum size because it has not been backed up and truncated. In MySQL, the transaction log (often the InnoDB redo log or binary log) must be backed up periodically to free space; otherwise, write operations fail. This is a database-side issue, not a Glue or permissions problem, so the correct action is to back up the transaction log to release space.

Answer

The MySQL database has reached its maximum number of concurrent connections.

Answer

The AWS Glue job does not have sufficient permissions to write to the S3 bucket.

Answer

The AWS Glue job is configured with an incorrect 'writeDynamicFrame' method.

Question 16

A company runs a data pipeline that ingests clickstream data from a web application into Amazon Kinesis Data Streams. A Lambda function processes records from the stream and writes them to an Amazon S3 bucket in JSON format. The pipeline has been running smoothly, but for the past hour, the Lambda function has been failing with 'Rate exceeded' errors, and the Kinesis stream shows elevated 'IteratorAgeMilliseconds' metrics. The Lambda function has a reserved concurrency of 100, and the Kinesis stream has 10 shards. The average record size is 5 KB, and the data rate is approximately 15 MB per second. Which combination of actions should a data engineer take to resolve the issue and prevent recurrence? (Choose TWO.)

Accepted Answer

Increase the Lambda function's reserved concurrency to 200.. The 'Rate exceeded' errors indicate that the Lambda function's concurrency is insufficient to keep up with the incoming data rate from Kinesis. With 10 shards and a 15 MB/s data rate, each shard processes ~1.5 MB/s, and with 5 KB records, that's ~300 records per second per shard. Increasing reserved concurrency to 200 allows more parallel invocations to handle the load, reducing the iterator age.

Answer

Decrease the Lambda function's batch size from 100 to 50.

Answer

Enable S3 multipart upload for the Lambda function.

Answer

Replace the Lambda function with Amazon Kinesis Data Firehose to write directly to S3.

Question 17

A company uses AWS Glue to run ETL jobs that process data from Amazon S3 and write results to Amazon Redshift. The Glue job uses the JDBC connection to Redshift. Recently, the job has been failing intermittently with the error: 'java.sql.SQLException: [Amazon](500310) Invalid operation: INSERT has more expressions than target columns;' The Glue job writes to a staging table in Redshift before performing a merge into the final table. The staging table schema matches the source data. The error occurs only on some days and affects different columns each time. The data engineer suspects that the source data occasionally contains extra columns due to a schema drift in the upstream data producer. Which approach should the data engineer take to handle this issue robustly?

Accepted Answer

Use a Glue DynamicFrame and apply the resolveChoice method to make the schema consistent.. Option B is correct because Glue DynamicFrames can automatically handle schema drift using the `resolveChoice` method, which allows you to specify how to handle columns that appear inconsistently across records (e.g., making them null, casting to a common type, or dropping them). This directly addresses the intermittent error caused by extra columns in the source data without requiring manual schema updates or fragile conditional logic.

Answer

Skip any records that have extra columns by adding a conditional check in the Glue script.

Answer

Manually update the Redshift staging table schema whenever the source data changes.

Answer

Use a Glue DynamicFrame and apply the dropFields method to remove extra columns before writing.

Question 18

A data engineer is troubleshooting a Glue ETL job that reads from an S3 bucket and writes to a Redshift table. The job fails with a 'MemoryError' when processing a large dataset. Which TWO actions should the engineer take to resolve this issue? (Choose TWO.)

Accepted Answer

Increase the number of DPUs and set 'spark.sql.shuffle.partitions' to a higher value.. Option A is correct because increasing the number of DPUs (Data Processing Units) provides more memory and compute resources to the Glue job, directly addressing the MemoryError. Setting 'spark.sql.shuffle.partitions' to a higher value reduces the amount of data shuffled per partition, preventing out-of-memory errors during wide transformations like joins or aggregations.

Answer

Increase the number of DPUs and set 'coalesce(1)' in the script.

Answer

Decrease the number of DPUs and increase 'spark.shuffle.partitions'.

Answer

Set the 'RedshiftTempDir' parameter to a larger S3 bucket.

Question 19

A data engineer applies the above S3 bucket policy to an S3 bucket used by a Glue ETL job. The Glue job writes objects to the bucket. Which of the following is true about the behavior of the policy?

Accepted Answer

The policy denies all PutObject requests because the Allow and Deny statements are contradictory.. The correct answer is C because in AWS IAM policy evaluation, an explicit Deny always overrides any Allow. The policy has an Allow statement granting s3:PutObject for all principals, but a separate Deny statement explicitly denies s3:PutObject when the encryption condition is not aws:kms. Since the Deny applies to all PutObject requests (including those with no encryption or AES256), and the Allow does not include a condition to match only aws:kms, the Deny takes precedence and blocks all PutObject requests, making the policy effectively deny all PutObject operations.

Answer

The policy allows PutObject with aws:kms encryption because the Allow statement is broader.

Answer

The policy allows PutObject with no encryption because the Deny only applies to PutObject.

Answer

The policy allows PutObject with AES256 encryption and denies PutObject with aws:kms encryption.

Question 20

A company runs a nightly batch processing pipeline using AWS Glue ETL jobs. The pipeline reads data from an Amazon S3 bucket, transforms it, and writes results to an Amazon Redshift cluster. Recently, the data volume has increased significantly, and some Glue jobs are failing with the error 'java.lang.OutOfMemoryError: Java heap space'. The data engineer needs to modify the job configuration to prevent these failures without changing the code. The job currently uses 10 DPUs and processes data in a single Spark DataFrame. Which of the following is the MOST effective solution?

Accepted Answer

Increase the number of DPUs to 20 and enable job bookmarking for incremental processing.. Increasing DPUs from 10 to 20 provides more memory and compute resources, directly addressing the 'java.lang.OutOfMemoryError: Java heap space' caused by insufficient memory for the single DataFrame. Enabling job bookmarking allows incremental processing, which reduces the volume of data processed per run, further mitigating memory pressure without code changes.

Answer

Reduce the number of DPUs to 5 and increase the Spark executor memory by setting 'spark.executor.memory' in job parameters.

Answer

Change the script to use DynamicFrame instead of DataFrame and disable the 'spark.sql.shuffle.partitions' configuration.

Answer

Add a 'coalesce(1)' operation before writing to Redshift to reduce the number of output files.