Data Operations and Support practice questions

Increase the number of KCL workers per shard (e.g., 2 workers per shard).

Why wrong: The KCL does not support multiple workers per shard; each shard is processed by a single worker sequentially. Therefore, increasing workers per shard is not a valid configuration.

Increase the number of shards to 20.

Why wrong: Increasing the number of shards can help if the shard is saturated with incoming data, but the problem is consumer lag (processing speed), not ingestion throughput. Enhanced Fan-Out is more targeted.

Reduce the record size by compressing the data.

Why wrong: Compressing records reduces data size, which can improve throughput, but does not address the consumer's processing latency. The records are small (1 KB) already, so compression offers minimal benefit.

The JDBC connection string has incorrect credentials.

Why wrong: Incorrect credentials produce an authentication error, not a timeout.

The source database schema has changed.

Why wrong: Schema changes cause SQL parsing errors, not connection timeouts.

The Glue job's timeout setting is too low.

Why wrong: Job timeout is separate; the error is a connection timeout from the database side.

Use AWS SCT to convert the schema and then use DMS for full load

Why wrong: SCT is not for replication.

Use a full-load task that runs daily

Why wrong: Full load is not continuous and has high latency.

Use Kinesis Data Streams to capture changes and write to S3

Why wrong: Kinesis is not designed for direct database replication.

The query syntax is invalid

Why wrong: The syntax is correct.

The query limit of 20 is too low

Why wrong: A limit of 20 returns up to 20 results.

The time range is not specified and defaults to the last 15 minutes

Why wrong: Even if no time specified, it scans all logs; no results means no errors.

Increase the number of workers

Why wrong: Increasing the number of workers does not increase memory per worker, so it may not resolve the OutOfMemory error.

Switch to G.1X worker type

Why wrong: Switching to G.1X provides 1 DPU (8 GB) per worker, more memory than Standard but less than G.2X; not the most effective option.

Increase the number of DPUs per worker

Why wrong: Increasing DPU per worker is not a configurable parameter in AWS Glue; DPU allocation is determined by the worker type. This option is invalid.

s3:PutObject

Why wrong: Incorrect. s3:PutObject allows writing objects to the bucket, but the error is about listing, not writing.

s3:DeleteObject

Why wrong: Incorrect. s3:DeleteObject allows deleting objects, not listing the bucket.

s3:GetObject

Why wrong: Incorrect. s3:GetObject allows reading objects, not listing the bucket.

Decrease the batch size in the Flink application.

Why wrong: Smaller batches reduce throughput.

Switch the source stream to use GZIP compression.

Why wrong: Compression affects storage, not processing lag.

Increase the retention period of the Kinesis stream.

Why wrong: Retention period does not affect processing speed.

Use DynamoDB for checkpointing to track processed records.

Why wrong: Incorrect. DynamoDB checkpointing is typically used to track processed records for at-least-once or exactly-once processing, not for at-most-once.

Set the Lambda function's batch size to 100 to process records in larger batches.

Why wrong: Incorrect. Larger batch sizes increase the likelihood of partial failures and the need for retries, which is unsuitable for at-most-once.

Enable function-level retries in the Lambda function for transient errors.

Why wrong: Incorrect. Enabling function-level retries would cause record reprocessing upon failure, contradicting at-most-once semantics.

Increase the consumer's processing speed to reduce lag.

Why wrong: Consumer speed affects read throughput, not write throttling.

Reduce the data retention period of the stream.

Why wrong: Retention period does not affect write throughput.

Implement exponential backoff and retries in the producer applications.

Why wrong: Retries mitigate transient errors but do not increase capacity.

The IAM policy does not include the bucket ARN for write operations.

Why wrong: The policy includes the bucket ARN for ListBucket and object ARN for Get/Put.

The IAM role's trust policy does not allow Glue to assume the role.

Why wrong: A trust policy issue would cause an assume role error, not access denied on S3.

The IAM policy does not grant s3:PutObject permission.

Why wrong: The policy includes s3:PutObject.

The evaluation periods are set too low.

Why wrong: Low evaluation periods would still show data if metrics exist; they might cause missing data only if the metric was just created.

The alarm threshold is set too high.

Why wrong: Threshold affects the alarm state (OK or ALARM), not INSUFFICIENT_DATA.

The alarm does not have any actions configured.

Why wrong: Actions are not required for the alarm to evaluate data.

Change the source file format from Parquet to JSON.

Why wrong: Changing the file format from Parquet to JSON typically increases memory usage due to larger file sizes and lack of compression, worsening the MemoryError.

Reduce the number of partitions in the source data.

Why wrong: Reducing the number of partitions can lead to larger partitions, increasing memory pressure per task and potentially worsening the issue.

Use S3 Select to filter data before processing.

Why wrong: S3 Select can filter data before processing but does not directly resolve a MemoryError in a Glue job; it may reduce data volume but increasing DPUs is a more direct solution.

kms:Decrypt on a KMS key

Why wrong: If encryption is used, this might be needed, but the question does not mention encryption.

ec2:DescribeInstances on the Redshift cluster

Why wrong: Not required for Glue to write to Redshift.

lambda:InvokeFunction on a Lambda function

Why wrong: Lambda is not part of the Glue job in this scenario.

Set up a secondary Redshift cluster in another region and use AWS DMS for continuous replication.

Why wrong: AWS DMS continuous replication adds complexity and cost; while it can achieve low RPO/RTO, the operational overhead is higher than automated snapshot copy.

Export the data to S3 daily using UNLOAD and copy to another region.

Why wrong: Daily UNLOAD to S3 and cross-region copy results in an RPO of up to 24 hours, exceeding the 1-hour RPO requirement. Restore process is also slower, failing the RTO.

Enable automated snapshots with a retention period of 1 day.

Why wrong: Enabling local automated snapshots with 1-day retention does not provide cross-region redundancy. If the primary region fails, snapshots are lost, rendering the strategy ineffective for disaster recovery.

The S3 bucket is in a different AWS Region than the Glue job

Why wrong: S3 bucket region does not cause access denied errors if the IAM role has cross-region permissions. The error is about permissions.

S3 Server Access Logging is enabled and blocking requests

Why wrong: S3 Server Access Logging does not block requests; it only logs them.

The S3 bucket has S3 Transfer Acceleration enabled

Why wrong: S3 Transfer Acceleration is about speed and does not affect access permissions.

Change the worker type to G.1X

Why wrong: G.1X has less memory than G.2X.

Increase the number of workers in the Glue job

Why wrong: Does not reduce per-worker memory pressure.

Use a G.2X worker type with more memory

Why wrong: Helps but not as effective as reducing partitions.

Increase the number of core nodes in the EMR cluster.

Why wrong: Adding nodes increases total capacity but does not fix per-executor memory limits.

Decrease spark.sql.shuffle.partitions to reduce overhead.

Why wrong: Fewer partitions may cause larger partitions, worsening memory issues.

Increase spark.driver.memory in the Spark configuration.

Why wrong: Driver memory is for the driver process, not executors.

Run the VACUUM command on the source tables.

Why wrong: VACUUM reorganizes data but does not update materialized views.

Clear the Redshift result cache by running RESET ALL.

Why wrong: Result cache is for repeated queries, not for materialized views.

Run the ANALYZE command on the source tables.

Why wrong: ANALYZE updates statistics, not the data in materialized views.

Update the IAM role to grant additional S3 read permissions.

Why wrong: Permissions are not the issue; bookmark state is.

Reset the job bookmark using the AWS Glue API.

Why wrong: There is no API to reset a bookmark; you must disable it or change the job name.

Increase the number of workers in the Glue job.

Why wrong: Increasing workers improves parallelism but does not affect bookmarks.

The RDS instance has reached its maximum number of connections.

Why wrong: This would result in a 'too many connections' error, not connection refused.

The Glue job is using too many DPUs and hitting resource limits.

Why wrong: Resource limits would cause throttling or out-of-memory errors, not connection refused.

The IAM role associated with the Glue job lacks permissions to write to the S3 bucket.

Why wrong: This would cause an access denied error, not connection refused.