AWS Certified Data Engineer Associate DEA-C01 (DEA-C01) — Questions 226–300

Parallel scans can improve scan performance.

Kinesis Data Firehose ingests streaming records and delivers them to Amazon S3 with built-in buffering, compression and format conversion, requiring no consumer code. That satisfies real-time ingestion from thousands of IoT devices plus durable S3 storage for later analytics.

Apache Parquet is a columnar format that stores data by column, enabling Athena to read only the columns referenced in a query. This reduces I/O and cost. Parquet also supports predicate pushdown and is natively supported by Athena, making it ideal for this scenario where the goal is optimal query performance and minimal cost.

Converting to columnar Parquet with partitioning lets Athena read only needed columns and partitions, cutting bytes scanned. Registering the tables in the Data Catalog makes them immediately queryable without managing a separate metastore, and Glue handles the conversion job, keeping operational overhead low compared with self-managed ETL.

DynamicFrames natively accommodate schema evolution: resolveChoice reconciles columns that appear or disappear across files by casting or dropping them, while applyMapping standardises surviving fields before writing Parquet. This satisfies the inconsistent-schema constraint, and Glue job bookmarks let the same job process newly arrived files incrementally.

Concurrency scaling adds transient Redshift clusters that serve read queries, so eligible read workloads move off the main cluster and its CPU utilisation drops. This directly addresses the high CPU constraint by offloading read query processing rather than resizing or tuning the existing cluster.

Deleting an object in a versioned bucket inserts a delete marker rather than erasing prior versions. Removing that delete marker restores the object to its previous state instantly, with no data transfer and no re-upload, since the original version's bytes were never removed from the bucket.

AWS Glue jobs run inside a VPC only when attached to a subnet; without that attachment they cannot route to a private RDS instance. The S3 VPC endpoint does not provide a path to RDS, so the connection fails during the handshake.

An S3 bucket policy denying requests where aws:SecureTransport is false enforces TLS on every request, so Glue-to-S3 traffic cannot fall back to plain HTTP. This satisfies the in-transit encryption requirement at the bucket level, covering all clients including AWS Glue.

AWS Glue ETL jobs convert JSON to Parquet on serverless Spark infrastructure, eliminating cluster provisioning and satisfying the minimal-operational-overhead constraint. Parquet's columnar layout and compression reduce Redshift storage and scan costs, meeting the cost-effectiveness requirement for the nightly 10 MB file transformations.

The ClassNotFoundException for the Hive metastore class means the Hive client JARs are absent from the Python shell job's classpath. Adding the Hive JARs to the Python library path, or using a Glue version bundling Hive support, supplies the missing classes.

Job bookmarks rely on the transformation_ctx value to namespace state per source and sink; when it is absent or has changed between runs, Glue cannot correlate prior state and falls back to reading everything. Supplying a stable transformation_ctx on the S3 source and sink, then resetting the bookmark to clear stale state, restores correct incremental processing.

AWS Glue satisfies the hourly schedule constraint through time-based triggers, and its ETL engine converts JSON to Parquet using built-in transforms that infer schemas automatically. Crawlers catalogue the S3 source, while the Parquet output is written directly to S3 for efficient downstream querying by Athena or Redshift Spectrum.

AWS Glue ETL can transform JSON to Parquet, repartition by date, and update the Data Catalog, which Athena uses for schema and partition metadata. Parquet with Snappy reduces storage and scan size, and fewer larger files improve query performance. This directly addresses the small-file problem and columnar requirement while preserving date partitioning.

S3 SlowDown is a throttling response signalling too many concurrent requests to a partition. Exponential backoff with retries spaces requests progressively, letting the Glue job ride out the throttle rather than failing, which resolves the intermittent S3ServiceException errors.

CloudTrail records S3 data-plane and management API activity, capturing both successful and failed calls with identity, timestamp and source IP. This satisfies the compliance requirement to audit all bucket access, which S3 server access logging cannot match for failed calls.

Adding nodes increases both storage capacity and compute parallelism across the cluster, relieving the 80% disk pressure while distributing query workload. This addresses the root cause of degradation rather than temporarily masking it, providing a durable performance improvement for peak-hour analytics.

Athena, built on the AWS Glue Data Catalog, supports JSON SerDe and complex types such as struct, array, and map. Defining columns with these types lets SQL reference nested attributes directly, for example using dot notation on a struct field. No preprocessing is needed, which satisfies the requirement to query raw JSON in place while still allowing filters on individual attributes.

Partitioning S3 input by date lets Glue read only the relevant partitions, cutting the bytes scanned each run. This addresses the growing data volume constraint far more effectively than scaling workers or changing formats alone.

Amazon Kinesis Data Analytics for Apache Flink runs continuous SQL or Flink applications directly over the stream, joining each record against the DynamoDB reference table in-flight. This satisfies the real-time enrichment requirement with minimal latency, avoiding the batching delay of Lambda-based or Firehose-based approaches.

AWS Glue jobs writing to Amazon Redshift require the target table to exist beforehand. The error indicates the table is missing. The engineer must create the table in Redshift with a schema that matches the output of the Glue job, including column names and data types. This is a common prerequisite when using the Redshift connector in Glue Studio.

DMS ongoing replication reads the PostgreSQL write-ahead log via logical replication slots, capturing inserts, updates and deletes as they occur and streaming them to Kinesis. This satisfies the requirement to capture change data directly from the database without application changes.

Amazon Athena queries S3 data directly using standard SQL, reading the table and partition metadata held in the AWS Glue Data Catalog. It satisfies the SQL query requirement without loading data, complementing Glue's cataloguing role for the CSV, Parquet and image lake.

Spark's S3A filesystem lists the bucket or prefix before reading and writing objects, and that listing call requires s3:ListBucket on the bucket resource. GetObject and PutObject alone are insufficient, so the missing ListBucket permission causes the S3AccessDenied failure.

The aws:SourceVpc condition key evaluates the VPC endpoint through which the request arrives, so the bucket policy denies any traffic not originating from the specified VPC. This directly satisfies the requirement that the bucket be accessible only from that VPC.

The producer writes 8 MB/s while five shards provide only 5 MB/s aggregate, causing WriteProvisionedThroughputExceeded. Doubling to ten shards raises capacity to 10 MB/s, comfortably absorbing the sustained 8 MB/s load and clearing the error.

The Action element lists the specific API operations a policy allows or denies, so specifying dynamodb:GetItem and dynamodb:Query grants exactly those read-only calls and nothing else. This directly satisfies the stem's constraint of restricting access to only those two DynamoDB actions within the account.

Unevenly distributed write traffic concentrates load on a subset of partitions, so individual partitions exceed their 1,000 WCU per-partition limit even though total provisioned capacity of 12,000 WCU is not exhausted. DynamoDB throttles at partition level, making hot partitions the likely cause of the latency and throttling during peak hours.

Redshift dynamic data masking applies a masking policy to a column so that unauthorized users see a masked value at query time, while authorized roles see the full value. It is role-based and does not require changing the stored data. This directly satisfies the requirement to show partial values to most users and full values only to a specific role.

AWS Transfer Family provides a fully managed FTP endpoint that writes directly into Amazon S3, satisfying the daily CSV ingestion requirement without custom code or servers. Unlike AWS DataSync, which synchronises existing data stores, Transfer Family natively speaks FTP, letting the external partner push files straight to S3.

Full load plus ongoing replication (CDC) migrates existing rows and then continuously applies change data capture records from the Oracle redo logs, keeping S3 synchronised with minimal latency. A full-load-only task would leave subsequent changes unreplicated.

Format conversion performs schema-aware JSON-to-Parquet transcoding inside the Firehose delivery stream, using a referenced AWS Glue Data Catalog table to map source fields to Parquet columns. This satisfies the stem's requirement that records be transformed before delivery, removing the need for downstream ETL or Lambda-based conversion.

Lake Formation provides fine-grained access control and can be used to enforce anonymization policies.

Athena charges by data scanned, and columnar Parquet with predicate pushdown lets the engine read only the columns and row groups needed. Combined with partition pruning on year/month/day, queries skip irrelevant prefixes entirely. This directly reduces bytes scanned, lowering cost and improving latency without changing the query interface.

Enables real-time processing and schema evolution.

Enabling Flink metrics in CloudWatch exposes heap usage, garbage-collection pressure and checkpoint behaviour, revealing whether state growth or backpressure—not raw parallelism—causes the out-of-memory failures. Since TaskManager memory and parallelism were already raised without success, these per-operator metrics identify the actual bottleneck before further resource changes.

Sort keys physically order rows on disk by transaction_date, so Redshift's zone maps skip irrelevant blocks when filtering that column, cutting scanned data. This targets the stem's frequently filtered column directly, unlike distribution keys, which address join and data-skew costs instead.

DataBrew filter conditions are case-sensitive by default. If the source data contains 'invalid' rather than 'INVALID', a condition checking for 'INVALID' will not match, so those rows are retained. The engineer should either normalize case or adjust the condition. This is a common cause of filters appearing to have no effect when the job succeeds but output is unchanged.

S3 Lifecycle configuration applies transition rules that automatically move objects to Glacier Deep Archive once they cross the specified age threshold, here 90 days. It satisfies the stem's requirement for automatic, age-based archival without custom code, and transitions are billed per request rather than requiring retrieval or manual intervention.

For a Glue job to read SSE-KMS encrypted data, its IAM role must have permissions to use the KMS key for decryption. The required permissions are kms:Decrypt and kms:GenerateDataKey (for some operations). Attaching a policy with these actions on the specific key resource grants the necessary access. Without these permissions, the Glue job will fail with access denied errors.

Firehose invokes a Lambda function only when it is attached as the delivery stream's data transformation source, which enables the buffered records to be processed before delivery. Without this configuration, Firehose writes raw records straight to S3, so transformation never occurs.

Performance Insights identifies the specific SQL statements and wait events driving high CPU, so slow reporting queries can be targeted rather than guessed at. This diagnostic step precedes changes such as indexing or instance scaling, satisfying the requirement to act first on evidence.

ListBucket operates on the bucket resource and GetObject on the object resource, so both actions are required for listing and reading. An Allow effect grants the analyst read-only access without write permissions, matching the least-privilege requirement for listing objects and retrieving their content.

S3 Standard-IA suits infrequent access with immediate retrieval, satisfying the low-frequency, on-demand constraint. Its lower storage cost than S3 Standard meets cost-effectiveness, while replicating across a minimum of three Availability Zones preserves durability. Unlike Glacier classes, no retrieval delay applies, so JSON remains instantly available.

Lake Formation grants SELECT on the analytics tables, letting the data scientist query them through Athena while Lake Formation mediates access to the underlying S3 objects. Direct S3 access stays blocked because permissions are enforced at the catalog and table level, not through S3 bucket policies.

The partitionColumns parameter in AWS Glue's write_dynamic_frame method allows you to specify columns to partition the output by. When writing to S3, Glue will create a directory structure based on these columns, such as year=2023/month=01/day=01. This is the standard and efficient way to partition output data in Glue, improving query performance and reducing costs for downstream analytics.

When AWS Glue writes to Amazon Redshift using the COPY command, Redshift itself must access the S3 bucket. If the Redshift cluster's IAM role does not have the necessary S3 permissions, Redshift cannot read the data, resulting in a Spectrum Scan Error with S3 Access Denied. The Glue job's role permissions are separate from Redshift's role.

A low-cardinality partition key, such as deviceId with few unique values, concentrates read traffic on a small number of partitions. DynamoDB limits each partition to 3,000 RCUs, so even if total consumed capacity is under 100 RCUs, a single partition can exceed its local limit and throttle requests. This matches the symptom of throttling despite low aggregate usage.

Parquet is a columnar format that offers better compression and faster read performance compared to JSON. Converting the source data to Parquet reduces I/O and CPU overhead during the ETL job, significantly improving performance. This is a common best practice for AWS Glue jobs reading from S3, especially when the data is large. It directly addresses the slow read and transform steps.

Lake Formation supports column-level permissions, allowing fine-grained access control. By granting SELECT on sensitive columns only to the Cardiology role and on non-sensitive columns to other roles, the engineer enforces the requirement with minimal overhead. Lake Formation manages the permissions centrally, and no additional ETL or view creation is needed. This is the most efficient and secure approach.

Correct. AWS Glue ETL can perform schema-aware transformations, including redacting PII fields, and write to a target S3 bucket.

The policy grants kms:Decrypt but conditions it on the request arriving via the S3 service, so the principal can decrypt only through S3 GetObject calls. Direct KMS decrypt requests, or access via other services, are denied by the condition.

DynamoDB returns ThroughputExceededException when read requests exceed the table's provisioned or on-demand read capacity. Glue's parallel executors consume read capacity units rapidly, exhausting the table's limits. This directly satisfies the stem's constraint: the read rate from DynamoDB surpasses available capacity, causing throttling rather than a Redshift or Glue configuration fault.

Firehose requires both the bucket policy and its assumed IAM role to grant access. The bucket policy alone is insufficient; if the role lacks s3:PutObject, every delivery attempt returns AccessDenied, which matches the persistent errors despite the policy appearing correct.

Specifying a customer-managed KMS key at cluster creation makes Redshift encrypt all data at rest with that key, satisfying the requirement for customer-managed encryption. Redshift cannot retroactively swap the key type, so this must be set during provisioning.

An explicit Deny in the bucket policy overrides any Allow in the identity policy, producing Access Denied on s3:PutObject. Inspecting the bucket policy for a deny targeting GlueServiceRole identifies that blocking statement, satisfying the need to find the actual cause.

Amazon DynamoDB delivers consistent single-digit millisecond latency for key-value workloads, satisfying the burst IoT ingestion requirement. Its partition-based architecture scales horizontally without manual sharding. However, DynamoDB alone cannot run complex analytical queries, so it must pair with a separate analytics service to meet the reporting constraint.

SSE-KMS with a customer managed key encrypts objects at rest under a key the team controls, and enabling automatic rotation on that key satisfies the rotation requirement. SSE-S3 and AWS managed keys cannot provide customer-controlled rotation.

AWS Glue ETL can read JSON, transform to Parquet, and write partitioned data to S3. Parquet is columnar, reducing storage and improving Athena query performance. Partitioning by date further reduces data scanned. This directly addresses the need for compact columnar format and cost efficiency.

Standard-IA reduces storage cost for infrequent access.

Lake Formation supports column-level security through data filters. A data filter allows you to specify which columns are included or excluded. By granting SELECT on the table and then creating a data filter that includes only customer_id and purchase_amount, the role can access only those columns. This is the correct way to implement column-level access control in Lake Formation.

RDS does not support encrypting an existing unencrypted instance in place. Snapshot-copy with encryption re-encrypts the data under a customer-managed KMS key, and restoring creates a new encrypted instance, meeting the policy with only the brief downtime of a snapshot restore.

AWS Glue emits job state change events to Amazon EventBridge. You can create an EventBridge rule that matches Glue job state changes (e.g., FAILED, TIMEOUT) and targets a Lambda function or SNS topic directly. This is a serverless, operationally efficient approach that requires no polling and provides near-real-time alerts on job failures, including those caused by missing partitions.

Repartition or coalesce changes the number of partitions in the DataFrame, which directly controls how many files are written. Coalesce is especially useful when reducing partitions without a full shuffle. Applying it before the target write consolidates output into fewer, larger files and improves downstream query performance.

S3 Lifecycle policies define transition rules that automatically move objects to a lower-cost storage class once they reach a specified age, such as 365 days. This satisfies the stem's compliance requirement that data older than one year be moved automatically without manual intervention.

SSE-KMS encrypts objects using keys held in AWS KMS, and selecting a customer managed key means the customer generates and controls that key, including its policy and rotation. SSE-S3 uses AWS-owned keys the customer cannot manage.

Amazon Kinesis Data Firehose satisfies the near real-time ingestion and S3 delivery constraints by buffering streaming records and writing them directly to Amazon S3 without custom consumer code. It handles scaling, batching and format conversion automatically, so IoT telemetry lands in S3 for analytics with minimal operational overhead.

Firehose assumes the `firehose.amazonaws.com` service principal to call KMS GenerateDataKey and Decrypt on your behalf, so the CMK's key policy must explicitly grant that principal those actions. Without this grant, the key is inaccessible and delivery fails, even though the IAM role used for S3 access is correctly configured.

DynamoDB is a fully managed NoSQL database that supports high-throughput reads and writes with automatic scaling. It can handle millions of requests per second and provides low-latency performance. With features like time-to-live (TTL) and on-demand capacity mode, it is well-suited for time-series data and real-time analytics. Its ability to scale horizontally without downtime makes it the best fit for this scenario.

A SCHEDULED Glue trigger natively starts the job on a cron schedule, and setting the job's maximum concurrency to 1 prevents a new run from starting while a previous run is still active. Both settings live within Glue, so no external orchestration service is required. This is the simplest managed approach that meets the time-based schedule and the no-overlap requirement.

Raising spark.executor.memory enlarges each executor's JVM heap, directly addressing the 'OutOfMemoryError: Java heap space' thrown during Spark execution. Since the failure occurs in executors rather than the driver, this parameter targets the constrained component, giving shuffle and aggregation buffers sufficient headroom to complete the S3-to-HDFS job.

Firehose supports invoking a Lambda function to transform each record in flight before delivery, satisfying the stem's in-transit transformation requirement. The transformation occurs during ingestion rather than after landing, so downstream consumers receive already-processed data.

Partitioning the source data by columns used in filters allows Glue to read only relevant partitions, reducing I/O and the amount of data shuffled during joins. This can dramatically improve performance for large datasets. The engineer should ensure the Glue job uses predicate pushdown and that the partition columns are used in the query. This is a best practice for optimizing Glue ETL on S3.

DMS applies CDC changes using the source table's primary key to identify and update the correct target rows. Tables lacking a primary key cannot be matched reliably, so updates and deletes are dropped or misapplied, producing missing rows in the PostgreSQL target after replication.

DynamoDB supports customer managed KMS keys for encryption at rest. By specifying the key during table creation and granting DynamoDB permissions in the key policy, the table data is encrypted with that key. KMS key usage is logged in CloudTrail, providing the required auditability. This meets the security team's requirements.