CCNA Data Operations and Support Questions

Glue allocates memory per executor based on DPU count and worker type; a large in-memory dataset exceeding that allocation triggers OutOfMemoryError. Insufficient DPUs or an undersized worker type directly caps heap available to the job, so scaling them addresses the constraint stated in the stem.

VACUUM reclaims storage occupied by deleted rows and re-sorts unsorted data in Amazon Redshift, directly addressing rapidly growing disk usage. It is the specific maintenance command that returns freed blocks to the system for reuse.

Converting to Parquet reduces the amount of data scanned because Athena reads only the necessary columns, and partitioning by date allows Athena to skip irrelevant partitions. Running MSCK REPAIR TABLE updates the partition metadata so queries can leverage the partitions, significantly lowering cost and improving performance.

Enabling error logging for Firehose sends delivery errors to CloudWatch Logs, providing visibility into why records are not delivered. Common causes include insufficient S3 permissions, KMS key issues, or malformed records. Reviewing these logs allows the engineer to identify and fix the root cause, ensuring all records are delivered and reducing data freshness.

Lifecycle transition eligibility depends on the object's age, calculated as the current date minus LastModified. The CLI output supplies LastModified, so the current date is the missing value needed to determine whether the transition threshold has elapsed.

Lake Formation enforces its own table and column-level grants on top of IAM. The analyst's IAM policy permits Athena API calls, but without a Lake Formation SELECT grant on the table, the query is denied — explaining why the table appears in the catalog yet remains unqueryable.

Exponential backoff with jitter retries throttled DynamoDB requests, letting the table's provisioned capacity recover between attempts. This directly addresses the intermittent ProvisionedThroughputExceededException, which signals transient throttling rather than a persistent capacity shortfall. Retries absorb those spikes without over-provisioning, satisfying the requirement to stop hourly Glue job failures.

Without `logs:CreateLogGroup`, `logs:CreateLogStream` and `logs:PutLogEvents` in the execution role, Lambda cannot create the function's log group or stream, so invocations produce no CloudWatch output at all. This directly satisfies the stem's missing-logs constraint, since the trigger itself is firing normally.

AWS Glue requires that JDBC drivers be provided as additional jars via the 'Dependent jars path' and accessible from S3. Simply setting a connection parameter is insufficient. The job's IAM role must also have permission to read the jar from S3. This is the documented approach for using custom JDBC drivers with Glue.

Adding DPUs increases the total executor memory and parallelism available to the Glue job, letting Spark distribute transformation partitions across more workers. This directly addresses the OutOfMemoryError caused by insufficient memory per executor during the transformation phase.

Writing transformed output back to Amazon S3 requires s3:PutObject on the target prefix. Without it, the Glue job's write stage returns Access Denied even when reading succeeds, so this permission must be added to the job's IAM role.

The DLQ captures failed records so Kinesis retries do not discard them, while raising DynamoDB write capacity removes the throttling that triggers ProvisionedThroughputExceededException. Together they satisfy the no-data-loss constraint, though the DLQ alone would not fix the underlying capacity shortfall.

Amazon MWAA uses an execution role that grants the Airflow environment permissions to access AWS services. To allow MWAA to read and write to S3, you must attach an IAM policy to this execution role with the appropriate s3:GetObject, s3:PutObject, and s3:ListBucket permissions. This is the standard method for granting MWAA access to S3.

The 'groupFiles' option in AWS Glue ETL allows grouping multiple small files into a single partition, reducing the number of tasks and improving read efficiency. This directly addresses the issue of many small files causing overhead. By setting groupFiles to 'inPartition' or 'inPartitionAcrossBuckets', the job can process larger chunks of data per task, reducing garbage collection and improving performance.

Merging many small files reduces per-file overhead: Athena must open, list and read metadata for each object, so fewer larger files cut S3 request costs and task scheduling overhead. This directly addresses the small-file scanning inefficiency described in the stem.

The Glue crawler's schema-update behaviour controls whether re-crawls revise the Data Catalog table definition. Setting it to update the table schema lets the crawler add newly detected columns automatically, satisfying the requirement that schema changes in the Parquet data be picked up without manual intervention.

The groupFiles and groupSize options in AWS Glue allow the job to coalesce many small files into larger groups before processing. This reduces the number of input partitions and the memory overhead on the driver, directly addressing the OutOfMemory error and improving performance. It is the recommended approach for large numbers of small files without changing the output format.

The JDBC connector reads Redshift row-by-row through the driver, which is slow at this scale. Using the S3 staging option runs a Redshift UNLOAD to S3 in parallel, then Glue reads S3 directly, removing the JDBC bottleneck and cutting job duration.

CloudWatch Alarms can monitor AWS Glue job metrics such as glue.driver.aggregate.elapsedTime and glue.driver.aggregate.numFailedTasks. When a threshold is breached, the alarm can publish to an SNS topic, sending notifications. This requires minimal setup and no custom code, making it the least operational overhead solution for alerting on job failures and durations.

Millions of sub-1 MB files force Athena to open each object individually, so per-file overhead dominates and queries time out. Compacting into roughly 128 MB objects cuts the number of GET requests and partition listings, letting Athena scan far fewer, larger files.

Amazon CloudWatch captures Glue job metrics and emits events on state changes such as FAILED, satisfying the requirement to detect job failures. Combined with Amazon SNS for notification delivery, it forms the monitoring half of the pair. Glue natively publishes logs and metrics to CloudWatch, so no custom instrumentation is needed.

Core nodes hold HDFS data and shuffle output, so exhausted local disk blocks task execution. Expanding each core node's attached EBS volume increases available block storage, directly resolving the disk-space constraint without altering cluster topology or losing HDFS data.

Redrive allows resuming a failed execution from the point of failure, preserving previous successful steps and their outputs. This avoids re-running the entire workflow and is the recommended way to recover after correcting the root cause. It requires the state machine to have the necessary IAM permissions and the execution to be in a failed state.

The Step Functions task timed out because its TaskTimeout was set to 900 seconds, but the Glue job took 18 minutes (1080 seconds). The optimized Glue service integration waits for the job to finish, and if the task exceeds the TaskTimeout, Step Functions fails the task even if the Glue job continues running. Increasing the TaskTimeout to a value larger than the job's maximum runtime, such as 3600 seconds, allows Step Functions to wait for completion without timing out.

Security groups are stateful, but the Glue job's own outbound rules must still permit traffic to the RDS security group on MySQL port 3306. Inbound access alone is insufficient, so the missing egress rule explains the communications link failure.

The job bookmark key is used to identify the bookmark state. If the key changes, the job treats it as a new job and starts processing from the beginning, causing duplicates. Ensuring the bookmark key remains consistent is crucial for proper bookmark functionality.

The AnalysisException arises because Spark resolves column names against the schema registered in the Data Catalog, not the actual S3 objects. If the source's schema evolved—columns renamed, added or dropped—without a crawler or manual update refreshing the catalog table, the job's query references columns the catalog no longer exposes.

Grouping coalesces many small S3 objects into larger input partitions, so Glue reads far fewer files and holds less per-file metadata and buffer overhead in the driver and executors. This directly relieves the Java heap exhaustion caused by the large number of small files described in the stem.

Glue's JDBC connection requires the MySQL driver JAR to be supplied via the job's dependent JARs path; the bundled environment lacks it, so the driver class cannot load and the connection throws 'No suitable driver'. Adding the MySQL Connector/J JAR resolves the missing driver.

The job is bottlenecked by compute, not partitioning or file size. Raising Max capacity adds more DPUs, distributing the Parquet transformation across additional executors and reducing wall-clock time, directly addressing the stem's constraint that the job runs on the default DPU allocation.

Correct. The IAM role assigned to the Firehose delivery stream must have the s3:PutObject permission to write objects to the S3 bucket. The bucket policy allowing the service principal is not sufficient; the role also needs the appropriate S3 action.

DISTSTYLE KEY distributes rows according to a chosen column's values, so matching join keys co-locate on the same slice. For frequently joined tables, this eliminates cross-node data movement during joins, satisfying the requirement to minimise data movement and speed queries.

Scaling DPUs adds parallel executors, so the 500 GB shuffle and Redshift write are processed concurrently rather than serially. With 10 DPUs the job is compute-bound in standard mode; doubling to 20 directly halves wall-clock duration, satisfying the requirement to most reduce job duration.

Validating records as they land in Amazon S3 catches malformed, incomplete or out-of-range data before Glue transforms it, preventing corrupt rows from propagating into Redshift. This satisfies the requirement to ensure data quality across the multi-source ingestion pipeline at its earliest stage.

VACUUM reclaims space from deleted rows and re-sorts data, while ANALYZE updates table statistics used by the query planner. Both address the high disk usage and slow queries without adding nodes, making them the correct first action.

In AWS DMS, table mappings define which tables are included or excluded from the migration. If some tables are missing, it is likely that they are not covered by the selection rules. Reviewing and updating the table mappings to include the missing tables will ensure they are replicated. This is a common oversight when setting up DMS tasks.

Without s3:ListBucket permission, the crawler cannot enumerate prefixes under the bucket, so newly written year/month/day partitions remain invisible even though the hourly schedule runs. The missing IAM action is the specific cause of undiscovered partitions.

Step Functions Retry is configured per state, so adding it to the specific Glue job task retries only that task up to three times. Exponential backoff is achieved with IntervalSeconds and BackoffRate, and ErrorEquals can include States.TaskFailed to catch transient failures. This precisely satisfies the requirement without affecting other jobs in the sequence.

Ongoing replication with CDC lets AWS DMS continuously apply source Oracle changes to Redshift, satisfying the requirement to keep Redshift synchronised with minimal latency after the initial full load. Without CDC, only a one-time migration occurs, so later source changes never propagate.

S3 Event Notifications fire on object creation, letting a Lambda verify the object is non-zero before relocating it to a ready prefix. The ETL job then reads only that prefix, so it never encounters the zero-byte partial writes.

DynamoDB Accelerator is an in-memory cache that serves eventually consistent reads in microseconds, absorbing the repeated read traffic that drives latency on the on-demand table. Placing DAX in front reduces read latency by orders of magnitude without changing the table's capacity mode.

groupByKey shuffles all values for a key before aggregation, which can produce enormous intermediate data. reduceByKey and aggregateByKey perform map-side combiners that pre-aggregate values locally before the shuffle, dramatically shrinking the data moved across the network. Less shuffled data means smaller per-executor memory footprints during the aggregation, addressing the root cause of the container kills.

AWS DMS logs errors for issues like unsupported data types, DDL changes, or transformation failures in CloudWatch Logs. These errors can cause specific transactions to be skipped during CDC. Reviewing the logs is the first step to identify why transactions are missing. The logs provide detailed messages that help pinpoint the root cause, such as a data type not supported by the target.

SSE-KMS encrypts objects at rest using keys managed in AWS KMS, satisfying the encryption-at-rest requirement. It provides envelope encryption with an independent data key per object, plus audit trails via CloudTrail and granular access control through key policies — unlike SSE-S3, which offers no separate key permissions or key-usage auditing.

Concurrency scaling adds transient Redshift clusters that serve read queries, so eligible read workloads move off the main cluster and its CPU utilisation drops. This directly addresses the high CPU constraint by offloading read query processing rather than resizing or tuning the existing cluster.

AWS Glue jobs run inside a VPC only when attached to a subnet; without that attachment they cannot route to a private RDS instance. The S3 VPC endpoint does not provide a path to RDS, so the connection fails during the handshake.

The ClassNotFoundException for the Hive metastore class means the Hive client JARs are absent from the Python shell job's classpath. Adding the Hive JARs to the Python library path, or using a Glue version bundling Hive support, supplies the missing classes.

Job bookmarks rely on the transformation_ctx value to namespace state per source and sink; when it is absent or has changed between runs, Glue cannot correlate prior state and falls back to reading everything. Supplying a stable transformation_ctx on the S3 source and sink, then resetting the bookmark to clear stale state, restores correct incremental processing.

S3 SlowDown is a throttling response signalling too many concurrent requests to a partition. Exponential backoff with retries spaces requests progressively, letting the Glue job ride out the throttle rather than failing, which resolves the intermittent S3ServiceException errors.

Adding nodes increases both storage capacity and compute parallelism across the cluster, relieving the 80% disk pressure while distributing query workload. This addresses the root cause of degradation rather than temporarily masking it, providing a durable performance improvement for peak-hour analytics.

DMS ongoing replication reads the PostgreSQL write-ahead log via logical replication slots, capturing inserts, updates and deletes as they occur and streaming them to Kinesis. This satisfies the requirement to capture change data directly from the database without application changes.

Spark's S3A filesystem lists the bucket or prefix before reading and writing objects, and that listing call requires s3:ListBucket on the bucket resource. GetObject and PutObject alone are insufficient, so the missing ListBucket permission causes the S3AccessDenied failure.

The producer writes 8 MB/s while five shards provide only 5 MB/s aggregate, causing WriteProvisionedThroughputExceeded. Doubling to ten shards raises capacity to 10 MB/s, comfortably absorbing the sustained 8 MB/s load and clearing the error.

Full load plus ongoing replication (CDC) migrates existing rows and then continuously applies change data capture records from the Oracle redo logs, keeping S3 synchronised with minimal latency. A full-load-only task would leave subsequent changes unreplicated.

Enabling Flink metrics in CloudWatch exposes heap usage, garbage-collection pressure and checkpoint behaviour, revealing whether state growth or backpressure—not raw parallelism—causes the out-of-memory failures. Since TaskManager memory and parallelism were already raised without success, these per-operator metrics identify the actual bottleneck before further resource changes.

Performance Insights identifies the specific SQL statements and wait events driving high CPU, so slow reporting queries can be targeted rather than guessed at. This diagnostic step precedes changes such as indexing or instance scaling, satisfying the requirement to act first on evidence.

When AWS Glue writes to Amazon Redshift using the COPY command, Redshift itself must access the S3 bucket. If the Redshift cluster's IAM role does not have the necessary S3 permissions, Redshift cannot read the data, resulting in a Spectrum Scan Error with S3 Access Denied. The Glue job's role permissions are separate from Redshift's role.

An explicit Deny in the bucket policy overrides any Allow in the identity policy, producing Access Denied on s3:PutObject. Inspecting the bucket policy for a deny targeting GlueServiceRole identifies that blocking statement, satisfying the need to find the actual cause.

AWS Glue emits job state change events to Amazon EventBridge. You can create an EventBridge rule that matches Glue job state changes (e.g., FAILED, TIMEOUT) and targets a Lambda function or SNS topic directly. This is a serverless, operationally efficient approach that requires no polling and provides near-real-time alerts on job failures, including those caused by missing partitions.

S3 Lifecycle policies define transition rules that automatically move objects to a lower-cost storage class once they reach a specified age, such as 365 days. This satisfies the stem's compliance requirement that data older than one year be moved automatically without manual intervention.

Amazon Kinesis Data Firehose satisfies the near real-time ingestion and S3 delivery constraints by buffering streaming records and writing them directly to Amazon S3 without custom consumer code. It handles scaling, batching and format conversion automatically, so IoT telemetry lands in S3 for analytics with minimal operational overhead.

A SCHEDULED Glue trigger natively starts the job on a cron schedule, and setting the job's maximum concurrency to 1 prevents a new run from starting while a previous run is still active. Both settings live within Glue, so no external orchestration service is required. This is the simplest managed approach that meets the time-based schedule and the no-overlap requirement.

Raising spark.executor.memory enlarges each executor's JVM heap, directly addressing the 'OutOfMemoryError: Java heap space' thrown during Spark execution. Since the failure occurs in executors rather than the driver, this parameter targets the constrained component, giving shuffle and aggregation buffers sufficient headroom to complete the S3-to-HDFS job.

DMS applies CDC changes using the source table's primary key to identify and update the correct target rows. Tables lacking a primary key cannot be matched reliably, so updates and deletes are dropped or misapplied, producing missing rows in the PostgreSQL target after replication.

Spark executor memory is set by spark.executor.memory; the OutOfMemoryError arises because the new EMR release changed default executor sizing, leaving too little heap. Raising this value directly expands executor heap, satisfying the job's memory constraint without altering cluster hardware.

Glue's schema inference can fail if CSV files have inconsistent columns, missing headers, or if the 'withHeader' option is not set to true. In this scenario, the header row exists but the default reader may not treat it as a header, leading to type conflicts. Setting 'withHeader' to true and ensuring consistent data resolves the error.

Without an S3 event notification targeting the Lambda function, uploads never generate an invocation, so no CloudWatch execution logs appear. Confirming the notification configuration first isolates whether the trigger exists before investigating permissions or code.

Glue assumes its execution role via AWS Security Token Service, which validates the role's trust policy. If glue.amazonaws.com is absent as a trusted principal, AssumeRole is denied, producing the 'Unable to locate credentials' error rather than a permissions failure.

Setting a WLM timeout on the long-running queue prevents those queries from monopolizing resources indefinitely. When a query exceeds the timeout, it is terminated, freeing up resources for other queries. This helps maintain throughput and ensures that short queries are not starved. It is a key configuration in manual WLM to manage runaway queries.

Correct. AWS Glue triggers allow you to schedule jobs using cron expressions. You can create a trigger of type SCHEDULED, set the cron expression to '0 2 * * ? *' for 2:00 AM UTC daily, and attach it to the Glue job. This is the simplest and most native way to schedule Glue jobs, with built-in monitoring through Glue job run metrics and CloudWatch.

The Retry field in a Step Functions task state allows you to specify retry behavior for failed tasks. Setting MaxAttempts to 3 and BackoffRate to 2.0 will retry the Glue job up to three times with exponential backoff (doubling the wait time between retries). This matches the requirement.

Memory-optimised r5.xlarge instances provide a higher memory-to-vCPU ratio than m5.xlarge, directly relieving the near-90% memory utilisation that is throttling Spark executors. Since CPU sits below 20%, the bottleneck is memory capacity, not compute, so swapping core nodes to r5.xlarge lets executors hold larger partitions without spilling to disk.

Short query acceleration (SQA) uses machine learning to predict query execution time and runs short queries in a dedicated space, preventing them from waiting behind long-running queries. Enabling SQA can improve overall throughput and reduce latency for short queries during peak periods, making it an effective action to resolve WLM-related performance issues.

The COPY command loads data in parallel from Amazon S3 into Redshift, directly satisfying the requirement to minimise cutover downtime during the full load. Its massively parallel architecture ingests the 500-million-row tables far faster than row-by-row INSERT statements, compressing the migration window before incremental updates begin.