Data Ingestion and Transformation practice questions

Increase the number of shards in the Kinesis stream.

Why it fails: Increases cost and may not be necessary.

Increase the Parallelization Factor for the Lambda event source mapping.

Why it fails: May increase concurrency but not cost-effective.

Increase the memory allocated to the Lambda function.

Why it fails: Does not reduce invocation rate.

Use S3 Batch Operations to write data in batches.

Why it fails: S3 Batch Operations performs bulk jobs on existing objects and cannot be invoked per streaming record inside Lambda; the throttling stems from per-prefix PUT request rates. Batch Operations would be correct for large-scale copying, tagging or deleting of already-stored S3 objects.

Increase the number of shards in the Kinesis data stream.

Why it fails: Increasing Kinesis shards raises ingestion capacity, but the failures occur on the Lambda-to-S3 write path, where ProvisionedThroughputExceededException signals S3 request-rate throttling per prefix. Shard count cannot relieve that. Adding shards is the right remedy when a stream's incoming records hit per-shard read/write limits.

Enable S3 Transfer Acceleration on the destination bucket.

Why it fails: Transfer Acceleration speeds uploads over long geographic distances via edge locations; it does not raise S3 request-rate limits, so per-prefix PUT throttling persists. It is tempting because it addresses S3 write performance, but it would be correct only for distant clients uploading large objects, not for exceeding per-prefix request rates.

Convert the CSV files to JSON format before processing.

Why it fails: JSON is a row-oriented text format that Glue still parses row by row, so it neither reduces heap pressure nor changes how Spark materialises partitions. It is tempting because JSON handles nested or irregular records well, making it the right choice when source schemas vary, not when executor memory is exhausted.

Decrease the Spark shuffle partitions in the job script.

Why it fails: Fewer shuffle partitions means larger, fewer tasks, concentrating more data into each executor and worsening heap exhaustion. It is tempting because reducing partitions cuts shuffle overhead and task scheduling on small datasets, which is the correct tuning when jobs spawn excessive tiny tasks rather than when executors run out of memory.

Increase the job timeout setting.

Why it fails: The timeout only governs how long a job may run before termination; it cannot relieve the heap pressure that triggers the OutOfMemoryError. It is tempting because raising it rescues jobs that fail purely by exceeding their allotted runtime, which is the genuine remedy for long-running jobs that complete successfully given more time.

Disable checkpointing to avoid failures.

Why it fails: Disabling checkpointing removes the fault-tolerance mechanism entirely, so the application cannot recover state after failure and reprocessing worsens. It is tempting because it eliminates the failing operation, but checkpointing is required for exactly-once and at-least-once guarantees in Flink.

Switch the state backend from in-memory to RocksDB.

Why it fails: RocksDB is already the default state backend in Kinesis Data Analytics for Apache Flink, so switching from in-memory changes nothing. It is tempting because RocksDB handles large state on disk, but the stem's intermittent checkpoint failures with large state point to checkpoint interval and timeout tuning instead.

Increase the parallelism of the application.

Why it fails: Increasing parallelism repartitions state across more subtasks, adding concurrent checkpoint writes and network overhead rather than improving reliability. It is tempting because parallelism boosts throughput, but checkpoint failures with large state stem from checkpoint interval, timeout, and storage configuration, not operator count.

Increase the Spark shuffle partitions to 500.

Why it fails: Raising shuffle partitions to 500 forces excessive small tasks and scheduling overhead for this data volume, slowing the job. It is tempting because more partitions can relieve skew, and would be correct when a few large partitions bottleneck a much bigger shuffle.

Write the output as JSON instead of Parquet to avoid compression overhead.

Why it fails: JSON is text-based and bulkier than Parquet, so writing it increases storage and read costs while abandoning columnar compression. It is tempting because JSON avoids schema handling, and would be correct when downstream consumers require semi-structured records.

Enable compression on the Kinesis stream.

Why it fails: Compressing the Kinesis stream reduces network bytes, but the Glue job still deserialises every JSON record into memory, so the out-of-memory condition persists. Compression suits lowering storage and transfer costs, not reducing executor memory pressure during processing.

Change the output format from Parquet to ORC.

Why it fails: ORC and Parquet are both columnar formats written after records are processed, so the out-of-memory failure during JSON reading is unaffected. Switching formats suits read-performance or storage-efficiency tuning on the output, not executor memory exhaustion.

Reduce the streaming batch size in the Glue job configuration.

Why it fails: Shrinking the batch size lowers per-microbatch memory pressure but does not address the actual cause: skewed partitions or unbounded state accumulating across the job. It is tempting because batch tuning genuinely helps when records are large and uniformly distributed, but here the OOM stems from state growth, which batch size cannot cap.

Amazon Kinesis Data Streams with AWS Lambda for transformation and Amazon S3.

Why it fails: Kinesis Data Streams with Lambda lacks native checkpointing of late records and exactly-once semantics, so replays or duplicates occur and hour-late data is mishandled. It is tempting because Kinesis with Lambda is the standard pattern for low-latency streaming ingestion where ordering and at-least-once delivery suffice.

Amazon Simple Queue Service (SQS) with AWS Lambda for transformation and Amazon S3.

Why it fails: SQS provides at-least-once delivery with no ordered replay window, so Lambda retries duplicate records and hour-late data cannot be reprocessed deterministically. It is tempting because SQS with Lambda is the classic decoupled queue pattern for buffering bursts and absorbing traffic spikes without provisioning stream shards.

AWS Glue streaming jobs consuming from Amazon Kinesis Data Streams and writing to Amazon S3.

Why it fails: Glue streaming jobs require deploying and managing a persistent Spark streaming job, including custom checkpointing logic to achieve exactly-once semantics. Kinesis Data Firehose is a fully managed delivery stream with built-in Parquet conversion, partitioning, and buffering — no cluster to provision or manage.

Increase the job frequency to every 30 minutes.

Why it fails: Running more often does not change bookmark precision, so sub-second records within the same second still get skipped. Increasing frequency is tempting because it reduces the window of missed data, but the root cause is timestamp granularity, not scheduling interval.

Run a full refresh of the table each time instead of incremental.

Why it fails: Full refresh reprocesses every row hourly, which is costly and slow but does not fix bookmark comparison at second precision. It is tempting because it guarantees no record is skipped, and it suits small tables where complete reloads are acceptable.

Modify the MySQL table to use a DATE data type instead of TIMESTAMP.

Why it fails: Changing the column to DATE discards the time-of-day entirely, so hourly incremental reads cannot distinguish records within the same day and would still miss or duplicate rows. DATE suits calendar-only values such as birth dates, where no time component is ever queried or compared.

Configure the crawler to 'Inherit schema from table' and set the table name.

Why it fails: Inheriting a schema copies an existing table's columns, so the crawler never inspects the CSV headers and cannot discover the extra columns. It suits deliberately pinning a known, fixed schema to a table, not the automatic detection this scenario requires.

Configure the crawler to 'Create a single schema for each S3 path' without enabling 'Merge tables'.

Why it fails: Forcing one schema per S3 path without merging makes the crawler pick a single file's header, so extra columns in other files are dropped rather than reconciled. It fits uniform files sharing an identical schema, not files whose columns diverge.

Configure the crawler to 'Create a single schema for each S3 path' and set 'Each file as a separate table'.

Why it fails: Setting each file as a separate table creates one catalog table per CSV, so no unified schema spanning the differing columns ever emerges. It suits querying heterogeneous files independently, whereas the engineer wants a single table with all columns detected.

Use the 'mergeSchema' option when reading the DynamicFrame.

Why it fails: Incorrect. `mergeSchema` is only available for Parquet/ORC formats, not CSV. Using it with CSV will have no effect on schema mismatches.

Convert all CSV files to Parquet format using a separate preprocessing job.

Why it fails: Incorrect. Converting to Parquet requires an extra preprocessing job, which is less efficient and adds complexity.

Set the job to 'ignore' schema mismatches in the job parameters.

Why it fails: No Glue job parameter silently ignores column-count mismatches; the crawler or job still fails schema detection. It is tempting as a quick unblock, and would fit if the requirement were merely to skip malformed records rather than reconcile differing schemas.

Enable GZIP compression on the Firehose delivery stream.

Why it fails: GZIP compression shrinks payload size but adds CPU work before delivery, so it does not cut the buffer-fill latency below 60 seconds during spikes. It is tempting because compression lowers S3 storage and transfer costs, and would be the right choice when optimising storage footprint rather than delivery latency.

Increase the buffer size to 128 MB to accommodate larger batches.

Why it fails: Raising the buffer to 128 MB lets Firehose wait longer before flushing, which increases delivery latency and breaches the 60-second requirement. It is tempting because larger buffers absorb volume spikes, and would be correct when throughput and fewer, larger S3 objects matter more than latency.

Switch to Kinesis Data Streams with a Lambda consumer.

Why it fails: Kinesis Data Streams with a Lambda consumer requires custom consumer code and checkpointing, adding operational overhead and latency versus tuning Firehose's buffer interval. It is tempting because Data Streams offers sub-second processing, and would be correct when multiple consumers must replay the same records.

Enable job bookmark and schedule the job to run more frequently.

Why it fails: Bookmarks only skip previously processed data; the initial 500-million-row load still runs over six hours, and more frequent runs add overlap. Bookmarks suit incremental ingestion once a baseline exists, not cutting a full extraction's duration.

Use multiple JDBC connections in parallel by setting 'hashexpression' and 'hashfield'.

Why it fails: Hash partitioning parallelises reads across multiple JDBC connections, but PostgreSQL must still serve every row, and the source is not the bottleneck being addressed by DPU scaling. It is correct when a single connection throttles throughput on a partitioned table.

Increase the number of DPUs for the Glue job to 100.

Why it fails: Adding DPUs scales Glue workers, but a single JDBC connection to PostgreSQL remains the throughput ceiling, so duration barely improves. It is tempting because DPU increases are the usual Glue scaling lever, and they work when the bottleneck is transformation, not extraction.

Increase the number of shards in the Kinesis stream.

Why it fails: Adding shards raises Kinesis ingest and read capacity, but the throttling originates from Lambda's DynamoDB writes, not stream throughput. Sharding is right when consumers hit per-shard read limits or producers exceed write capacity, which is not the bottleneck here.

Decrease the batch window in the Lambda event source mapping.

Why it fails: Shortening the batch window reduces how many records Lambda aggregates per invocation, producing more frequent, smaller DynamoDB writes and worsening throttling. A longer window is what batches writes; a short window suits latency-sensitive, low-volume workloads.

Increase the Lambda reserved concurrency to 1000.

Why it fails: Reserved concurrency sets a limit but does not prevent throttling if the account limit is exceeded.

Disable logging for the DMS task to free memory.

Why it fails: Disabling task logging frees negligible memory; OutOfMemory stems from buffering large BLOB rows, not log overhead. It is tempting because logging consumes some resources, but turning off logging is used to reduce write latency or CloudWatch costs, not to resolve LOB memory pressure.

Enable Full LOB mode to handle LOBs more efficiently.

Why it fails: Full LOB mode loads entire LOB values into memory, worsening OutOfMemory with large BLOBs; Limited LOB mode with an appropriate size cap is the memory-safe setting. Full LOB mode suits small LOBs where complete fidelity matters and memory is ample.

Change the Kinesis Data Analytics application to use a Kinesis Data Firehose delivery stream as the source.

Why it fails: Firehose is a delivery service, not a replayable stream source for Kinesis Data Analytics, so it cannot feed the application. It is chosen when buffering and loading data into S3, Redshift or OpenSearch, not for real-time SQL aggregation over a stream.

Reduce the retention period of the Kinesis Data Streams to 24 hours.

Why it fails: Shortening retention to 24 hours does nothing to processing throughput and risks data loss if the application lags beyond that window. Reduced retention suits cost control on streams consumed promptly, not a backlog requiring faster parallel processing.

Increase the number of shards in the Kinesis Data Streams to 20.

Why it fails: Adding shards raises ingest capacity but leaves the Kinesis Data Analytics parallelism at 1, so the single subtask still processes every shard sequentially and iterator age keeps climbing. Resharding suits ingest-side hot shards, not a downstream operator bottleneck; scaling the application's parallelism is what consumes shards concurrently.

Increase the number of shards in the Kinesis data stream to 20 and keep the Lambda configuration unchanged.

Why it fails: Adding shards raises parallel read capacity, but each Lambda invocation still runs with 128 MB and a 5-second timeout, so per-record validation and DynamoDB writes remain too slow and records still time out. Shard scaling is the right lever when consumers are CPU-bound and under-provisioned, not when the function's own execution limit is the bottleneck.

Replace the Lambda function with a Kinesis Data Firehose delivery stream that writes directly to DynamoDB using a Lambda transformation.

Why it fails: Firehose buffers records before delivery and cannot write to DynamoDB as a destination; its Lambda transform also runs synchronously per batch, so the 5-second timeout persists. Firehose suits streaming to S3, Redshift or OpenSearch, not low-latency DynamoDB writes requiring per-record enrichment at 1000 records per second.

Increase the Lambda function timeout to 60 seconds and memory to 1024 MB. Set the batch size to 100 records.

Why it fails: Raising timeout and memory addresses execution duration, but a batch size of 100 records at 2 KB each means each invocation processes roughly 200 KB sequentially, so throughput per shard still falls short of the incoming rate. Larger batches suit high-throughput batch consumers, not a stream needing low-latency per-record processing.

Enable the 'upsert' feature in the Redshift connection by setting 'update' to true.

Why it fails: Upsert requires a unique key; if not properly configured, duplicates can still occur.

Use partition pruning on the S3 source to reduce the number of files read.

Why it fails: Partition pruning reduces data scanned but does not prevent duplicate writes.

Increase the number of DPUs (Data Processing Units) allocated to the job.

Why it fails: More DPUs improve performance but do not prevent duplicates.

AWS Glue

Why it fails: Glue is for ETL jobs, not for file transfer.

Amazon AppFlow

Why it fails: AppFlow integrates with SaaS applications, not SFTP servers.

AWS DataSync

Why it fails: DataSync does not support SFTP as a source; it supports NFS, SMB, and self-managed object storage.

Set up administrators, then create S3 bucket, then register it in Lake Formation, then define schema in Glue Catalog, then grant access to users

Why it fails: This is incorrect because administrators cannot effectively manage the lake without a registered bucket; the bucket creation and registration should precede administrator setup.

Create S3 bucket, then define schema in Glue Catalog, then register it in Lake Formation, then set up administrators, then grant access to users

Why it fails: This is incorrect because the schema definition should happen after the bucket is registered in Lake Formation, as the schema is tied to the registered location; registering after schema definition complicates the setup.

Register S3 bucket in Lake Formation, then create S3 bucket, then set up administrators, then define schema in Glue Catalog, then grant access to users

Why it fails: This is incorrect because you cannot register a bucket that does not exist; the bucket must be created first before it can be registered in Lake Formation.

Run the query, then verify results, then create external table, then grant permissions, then create external schema.

Why it fails: This is incorrect because you cannot run a query without first having the schema and table defined and permissions granted. The order is completely reversed.

Create external table, then create external schema, then grant permissions, then run the query, then verify results.

Why it fails: This is incorrect because you must create the external schema before creating the external table; the table belongs to a schema.

Create external schema, then create external table, then run the query, then grant permissions, then verify results.

Why it fails: This is incorrect because you must grant permissions on the schema and table before running the query; otherwise, the query will fail due to lack of access.