AWS Certified Data Engineer Associate DEA-C01 (DEA-C01) — Questions 1051–1125

The CLI can filter by size and copy objects using a script.

DynamoDB divides a table into partitions, and each partition has a hard ceiling of roughly 3,000 read units and 1,000 write units per second regardless of the table's total provisioned capacity. If one key value, such as a default or shared customer ID, receives a burst of traffic, that partition saturates and throttles even though aggregate consumed capacity sits below the table's provisioned total.

Kinesis Data Streams ingests high-throughput streaming records with per-shard capacity that scales to thousands of producers, and delivers sub-second latency for real-time processing. It satisfies the 5 GB/min peak by adding shards, unlike S3 transfer or batch-oriented endpoints.

Job bookmarks track which S3 objects have already been processed and skip them on subsequent runs. For incremental data landing in S3, this avoids re-reading and re-transforming old files, directly reducing DPU hours and cost. It is a supported Glue feature and a common cost-control measure for recurring jobs.

Can be used for ETL jobs triggered by S3 events.

Amazon Managed Service for Apache Flink supports SQL-based stream processing on Kinesis Data Streams and can write results to Amazon S3, while the service manages the underlying infrastructure. It is designed for continuous, near-real-time transformations and removes the burden of provisioning and scaling servers, which matches the SQL-like, serverless-management requirement and the S3 output destination.

Consolidating many small S3 objects into fewer larger files via Glue ETL reduces the metadata and listing overhead the crawler processes each run. This directly addresses the small-file volume causing slow runs and timeouts, restoring crawler performance and reliability.

Cross-account access can be granted via bucket policies.

Kinesis Data Firehose can ingest data directly from Apache Kafka (via the Firehose HTTP endpoint or the Kafka connector) and deliver it to Amazon S3 without any custom code. It automatically handles scaling, buffering, and delivery, making it the most appropriate choice for minimal effort and no code.

More parallelism allows the application to process more records per unit time.

The AccessDenied error occurs on the ListBucket operation, which requires the s3:ListBucket permission on the bucket resource itself, not on objects. Adding this action to the Glue job's IAM role grants the missing listing capability.

Partitioning the data and using a columnar format like Parquet allows Glue to read only relevant partitions and leverage predicate pushdown. This reduces I/O and enables parallel processing across partitions. It directly addresses the slow performance caused by a single large file by splitting data into manageable chunks that can be processed concurrently.

Amazon Kinesis Data Firehose is a fully managed service that can ingest data from Kinesis Data Streams and deliver it to Amazon S3. It requires no code to set up and can buffer data for up to 60 seconds or based on size. It automatically handles scaling, and can convert JSON to Parquet or other formats if needed. This meets the requirements.

Configuring default encryption on the bucket applies SSE-KMS automatically to every object written, without relying on request headers. This enforces encryption at rest using AWS KMS as the stem demands, covering uploads from any client or SDK.

Distribution reduces data movement, sort key reduces data scanned.

Provisioned IOPS provides consistent low-latency writes.

Repartitioning on the join key before the join forces a shuffle that redistributes rows so that each partition contains a roughly equal share of data for that key. This directly addresses the skew that causes some executors to process far more rows than others. While a full shuffle has a cost, it is the standard remedy for skewed joins and can dramatically reduce overall job time when skew is severe.

Correct. Dynamic partitioning extracts partition keys from the data and creates S3 prefixes accordingly.

Using coalesce or repartition in the Spark code before the write step controls the number of output partitions, directly reducing small files. This preserves transformation logic while consolidating output, which is the standard remedy for too many small Parquet files in Glue jobs.

Glue is serverless and abstracts infrastructure, while EMR gives cluster-level control over instance types, libraries and dependencies. Needing custom execution environments or specific package versions therefore favours EMR, making control level a genuine selection factor.

The Map transform applies a user-supplied function to every record in a DynamicFrame and returns a new DynamicFrame with the transformed records. It preserves the one-to-one record relationship, making it ideal when each record is processed independently and the output schema is predetermined. It is the canonical Glue transform for this pattern.

A broadcast join is an optimization technique where the smaller table is replicated to all worker nodes, allowing the join to be performed locally without shuffling the large fact table. This reduces network overhead and improves performance for joins where one table is significantly smaller. AWS Glue supports broadcast joins when one side of the join is small enough to fit in memory.

On-demand mode removes provisioned WCU entirely, so DynamoDB instantly accommodates the 5000 WCU burst without throttling. Since the table is not auto-scaled and 1000 WCU is fixed, switching before the flash sale is the only action that absorbs the spike.

Parallel tasks allow concurrent loading of tables, increasing throughput.

Job bookmarks persist state from prior runs, so the JDBC source reads only rows added or changed since the last successful run rather than the full table. This cuts JDBC read volume, shuffle and S3 write costs, satisfying the growing-data-volume constraint while keeping the daily schedule.

Glue Studio provides a visual job editor with drag-and-drop nodes for sources, transforms, and targets. A visual job can implement column renames and filters without writing code, and Glue generates the underlying PySpark script automatically. This directly meets the requirement to minimize development time and avoid manual PySpark coding, while still running on the Glue Spark engine for scale.

Kinesis Data Firehose is a fully managed delivery stream that ingests streaming records and buffers them to S3 on a configurable interval, achieving near-real-time availability within minutes without consumer code. It matches the minutes-latency requirement for clickstream ingestion.

Increasing worker count directly addresses the 'Insufficient capacity' error, which occurs when Glue cannot provision enough DPUs to meet the job's requested executor count. Adding workers raises the total DPU allocation, allowing more Spark executors to launch and satisfy the pipeline's parallelism requirement.

Right-sizing matches provisioned cluster capacity to actual workload demand, eliminating spend on idle compute nodes. Redshift charges for running nodes regardless of utilisation, so analysing query concurrency, CPU and disk usage identifies over-provisioned clusters that can be resized to fewer or smaller nodes, directly reducing the cost driver described in the stem.

Job bookmarks persist state from prior runs and let the Glue job identify newly added S3 objects, so only incremental data is processed. This is the native, low-overhead mechanism for incremental processing in Glue ETL and directly solves the reprocessing cost problem without custom tracking.

Athena resolves partitions from the Data Catalog, not by listing S3 at query time. When Spark writes new partition prefixes without registering them, the catalog has no entries for those paths and the query planner filters them out, returning empty results for the new data. Registering the partitions with MSCK REPAIR TABLE or ALTER TABLE ADD PARTITION makes the new prefixes queryable.

On-demand capacity mode automatically scales shard throughput in response to traffic, removing the need to manually reshard as volume grows by 50%. It satisfies the no-manual-intervention constraint directly, unlike provisioned mode, which requires monitoring and explicit shard splits or merges to match demand.

Available network bandwidth is the limiting factor: at 1 Gbps, roughly 125 MB/s theoretical throughput caps the transfer, so 50 TB needs many hours regardless of DataSync tuning. Disk read speed, S3 request rates and agent CPU rarely saturate a 1 Gbps link, making bandwidth the primary determinant of total elapsed time.

Columnar formats such as Parquet or ORC store data by column, so Athena reads only the columns referenced in a query instead of every field. This directly cuts bytes scanned, which is the billed metric, lowering both query latency and cost for the existing Glue-produced data.

SSEAlgorithm AES256 indicates server-side encryption with Amazon S3-managed keys (SSE-S3), where S3 owns and rotates the AES-256 keys. This satisfies the stem's observed output directly: SSE-KMS would report aws:kms, and SSE-C would not appear as a bucket default algorithm.

Manually adding a partition to the AWS Glue Data Catalog when that partition already exists triggers HIVE_PATH_ALREADY_EXISTS, because Athena's metastore rejects duplicate partition locations for the year/month/day hierarchy. The error arises from the duplicate catalog entry, not from the underlying S3 data or query syntax.

The CloudTrail record shows the PutObject request carried the x-amz-server-side-encryption header set to aws:kms, confirming the object was written using SSE-KMS rather than SSE-S3 or SSE-C. This satisfies the investigation's need to identify the encryption mechanism applied to sensitive.csv at upload time.

Sort keys reduce the amount of data scanned.

DynamoDB export to S3 is a serverless, efficient way to export large tables without consuming read capacity. The export creates a consistent snapshot in S3, which Glue can then read in parallel. This method avoids impacting the table's provisioned throughput and is recommended for large-scale reads.

SSE-KMS encrypts objects with a KMS key, and decryption requires kms:Decrypt permission on that key. This satisfies the requirement for team-managed keys and explicit decrypt authorisation, unlike SSE-S3 where AWS owns the key and access is governed solely by S3 permissions.

Kinesis Agent tails log files, compresses, and sends to CloudWatch Logs; export to S3 can be automated.

A GSI allows querying on non-primary key attributes. By setting Category as the partition key and ProductID as the sort key, you can efficiently query all products in a category, sorted by ProductID. This is the optimal solution for this access pattern.

The 'Replace value or pattern' transform in DataBrew lets the engineer map each observed variant to a canonical value, handling 'US', 'usa', 'United States', and 'U.S.A.' in one recipe step. It matches the profiling output directly and produces a deterministic result, which is exactly what standardization requires before loading into Redshift.

Enforcing HTTPS via bucket policy satisfies the in-transit encryption mandate, while default SSE-S3 provides AES-256 server-side encryption at rest without key management overhead. S3 Inventory delivers scheduled CSV or Parquet reports of all objects and their metadata, meeting the compliance inventory requirement. Together these three controls cover every stated constraint.

S3 One Zone-IA stores data in a single Availability Zone, delivering millisecond retrieval while suiting infrequently accessed data. It satisfies the stem's latency constraint because retrieval remains immediate, unlike Glacier tiers requiring minutes to hours. Lower durability than Standard-IA is the trade-off, but the millisecond requirement is met.

Multi-AZ failover promotes the standby and repoints the DB instance's DNS CNAME to the new primary; clients caching the old record cannot connect until the TTL expires, producing the roughly 60-second outage described. This satisfies the stem's constraint that the endpoint itself was unreachable, not merely slow.

When JSON files have inconsistent structures, Glue's automatic schema inference can fail. Providing an explicit schema via a Data Catalog table or by passing a schema to the DynamicFrame resolves the ambiguity. This ensures the job can parse the data reliably regardless of variations, and is the recommended approach for production workloads with evolving or irregular JSON.

AWS IoT Core ingests high-volume, small-payload device telemetry and can route it directly to Amazon S3 via rules, meeting the 5 GB per hour near-real-time requirement. Its native MQTT support suits sub-1 KB events from a fleet, unlike services designed for batch or large-object transfer.

Amazon Kinesis Data Firehose is a fully managed service that can read from Kinesis Data Streams, transform data using AWS Lambda or built-in format conversion, and deliver to Amazon S3. It supports converting JSON to Parquet using the built-in record format conversion feature, which requires a AWS Glue table schema. This minimizes operational overhead and provides near real-time delivery.

S3 Object Lock in COMPLIANCE mode enforces a WORM retention period that no user, including the root account, can shorten or bypass, directly satisfying the requirement that objects cannot be deleted or overwritten for 7 years. GOVERNANCE mode would permit privileged users to alter retention, so it fails this constraint.

S3 Standard-IA is designed for infrequent access with millisecond retrieval.

DMS can write directly in Parquet format.

AWS Glue integrates with Secrets Manager through the connection's secretId property, letting the job fetch credentials at runtime. Granting secretsmanager:GetSecretValue scoped to the specific secret ARN provides least-privilege access. This avoids embedding plaintext credentials in scripts or job parameters while giving the job the password it needs.

Repartitioning or coalescing the DataFrame controls how many partitions exist at write time, and each partition produces one output file. Reducing partitions before the write consolidates output into fewer, larger Parquet files, which improves downstream query performance and directly addresses the many-tiny-output-files problem.

Increasing read capacity units raises the table's provisioned throughput ceiling, so reads are not throttled when consumed capacity approaches the limit during peak hours. For a workload dominated by single-item GetItem calls, this directly addresses the constraint causing latency: insufficient provisioned read throughput for the peak request rate.

Amazon RDS automated backups enable point-in-time recovery to any second within the retention period, up to 35 days. They are taken during a daily backup window and continuously archive transaction logs to S3, with minimal impact on the production database. Setting the retention period to 35 days directly meets the requirement.

Kinesis records carry a unique sequence number per shard, so using it as an idempotency key lets the Lambda function skip records it has already processed. This directly addresses duplicate writes at the sink with minimal code change, and it remains effective regardless of retries or throttling because reprocessed records are recognized and discarded.

Dynamic data masking in Amazon Redshift applies masking policies at query time, so PII columns return redacted values without altering stored data. Attaching these policies to roles satisfies GDPR's data-minimisation requirement, and unlike views, masking persists across all queries against the table, including ad-hoc analyst access.

Requesting a quota increase for concurrent executions directly addresses the throttling constraint: Step Functions limits how many state machine executions can run simultaneously per account and Region. When ETL jobs peak and hit that ceiling, new executions are rejected with 'Execution Throttled'. Raising the quota via Service Quotas lifts the ceiling, allowing the required parallelism.

Increasing shard count raises the stream's total write capacity, since each shard provides a fixed 1 MB/s and 1,000 records/s ingest limit. The WriteProvisionedThroughputExceeded error signals that producers are exceeding the aggregate provisioned throughput, so adding shards directly satisfies the peak-demand constraint described in the stem.

The Redshift COPY command loads CSV files directly from Amazon S3 into a table in parallel, parsing and distributing data across slices natively. It requires no cluster provisioning or transformation code, making it the simplest option for scheduled batch ingestion with matching schemas.

SSE-KMS encrypts objects with a KMS key, so s3:GetObject alone is insufficient: S3 must call KMS to unwrap the data key on the caller's behalf. The role therefore needs kms:Decrypt on that key, satisfying the stem's requirement to read the SSE-KMS-encrypted objects.

AWS Snowball Edge is a physical device used for offline data transfer. It can handle large volumes faster than network transfer and is cost-effective for this scenario.

DynamoDB global tables use multi-Region, active-active replication with a last-writer-wins conflict resolution, propagating writes from us-east-1 to the other Regions typically within a second. This directly satisfies the minimal-latency replication constraint, since replication is handled natively by DynamoDB rather than by custom streaming pipelines.

Kinesis Data Firehose can read directly from a Kinesis Data Stream, buffer records, and deliver them to S3. It supports record format conversion from JSON to Parquet using a schema defined in the AWS Glue Data Catalog. This requires no custom code and provides near real-time delivery, meeting all requirements with minimal effort.

Glue job bookmarks persist state about which S3 objects and partitions were processed in prior runs, so subsequent runs read only new data. This requires no change to the transformation logic and directly addresses incremental processing of newly arriving partitions.

DynamoDB's native export to Amazon S3 produces a full point-in-time snapshot in DynamoDB JSON or Amazon Ion format directly to an S3 bucket. It is serverless, requires no code, and does not consume provisioned or on-demand read capacity, making it ideal for a large nightly export within a tight window.

A Query operation targets a single partition key value and can filter the sort key with a range condition, so KeyConditionExpression on both keys returns exactly the user's items within the date range. Scan cannot filter on the sort key efficiently and reads the whole table.

The Flatten transformation in AWS Glue Studio un-nests nested structures such as arrays and structs into separate rows without writing custom code, which matches the requirement for a code-free, efficient transformation of nested JSON. It is designed exactly for this scenario and integrates with the visual job editor.

Adding DPUs scales the number of Apache Spark executors, giving more parallel workers to read the 50 GB DynamoDB table and write Parquet to S3. This directly addresses the compute-bound bottleneck, unlike tuning partition counts or connection settings.

Lake Formation data filters allow column-level and row-level security. Granting SELECT on the table with a data filter that specifies the allowed columns and a row filter expression restricts the analyst to only the permitted columns and rows. This meets the requirement precisely without granting broader access.

Kinesis Data Firehose offers a native record format conversion feature that transforms incoming JSON to Parquet using a schema from the AWS Glue Data Catalog, with no custom processing required. This directly satisfies the stem's requirement to convert JSON to Parquet before landing in S3, and remains the simplest option since Firehose handles the transformation itself.

Granting the partner account kms:Decrypt in the KMS key policy satisfies the stem's requirement that cross-account readers decrypt SSE-KMS objects. S3 bucket policies alone cannot authorise key usage; the customer-managed key's policy must separately permit the partner principal, since KMS enforces its own authorisation independently of S3.

DynamicFrame.from_options with format "json" and recurse=true handles nested and inconsistent JSON by flattening and inferring the schema dynamically, which suits semi-structured data without a catalog table. It is the standard programmatic approach in AWS Glue for reading JSON from S3 while accommodating schema variability at runtime.

The aws:SecureTransport condition key evaluates whether the request arrived over TLS; denying when it is false blocks any plain HTTP access to objects. This enforces encryption in transit between clients and the bucket, which is exactly the stem's stated requirement.