Data Ingestion and Transformation practice questions

AWS DataSync

Why wrong: For large data transfers between storage systems.

Amazon AppFlow

Why wrong: For SaaS data ingestion.

Amazon Kinesis Data Streams

Why wrong: Not the direct IoT ingestion service.

Switch from Spark to Python Shell job type.

Why wrong: Python Shell has limited memory.

Implement batch processing with smaller file sizes.

Why wrong: Requires code changes and may not be sufficient.

Use Redshift Spectrum to query data directly from S3.

Why wrong: Does not help Glue job memory.

The Kinesis stream's iterator age is exceeding the retention period.

Why wrong: Causes data loss, not duplicates.

The DynamoDB table used for checkpointing is throttling write requests.

Why wrong: Causes delays, not duplicates.

The record size exceeds the 1 MB API limit, causing retries.

Why wrong: Causes throttling, not duplicates.

AWS Glue DataBrew

Why wrong: For visual data preparation, not schema discovery.

AWS Lake Formation

Why wrong: For data lake access control.

Amazon Athena

Why wrong: Query service, does not discover schema.

Use Amazon Redshift COPY command to load JSON directly.

Why wrong: Does not transform to columnar format.

Use Amazon EMR with Spark to transform and load data.

Why wrong: Requires cluster management.

Use AWS Lambda to transform each file and write to Redshift.

Why wrong: Timeout and scalability issues.

Change the target format to JSON to reduce memory usage.

Why wrong: JSON is larger, not smaller.

Split the DMS task into multiple smaller tasks.

Why wrong: Requires major architecture changes.

Use Change Data Capture (CDC) only, without full load.

Why wrong: Does not address memory issue.

AWS Glue

Why wrong: ETL service, not direct ingestion.

AWS Database Migration Service (DMS)

Why wrong: For database replication.

Amazon Kinesis Data Firehose

Why wrong: For streaming data.

Increase the number of shards in the Kinesis stream.

Why wrong: Increases cost and may not be necessary.

Increase the Parallelization Factor for the Lambda event source mapping.

Why wrong: May increase concurrency but not cost-effective.

Increase the memory allocated to the Lambda function.

Why wrong: Does not reduce invocation rate.

Amazon SQS with AWS Lambda pollers to process tweets and store in S3.

Why wrong: Not designed for high-throughput streaming.

Amazon EMR with Spark Streaming to process tweets and write to S3.

Why wrong: Higher latency and management overhead.

Amazon API Gateway with AWS Lambda to process each tweet and store in S3.

Why wrong: May hit Lambda concurrency limits.

AWS DataSync

Why wrong: Too slow for 50 TB over 100 Mbps in one week.

Amazon CloudFront

Why wrong: Not for data transfer.

AWS Database Migration Service (DMS)

Why wrong: Not for HDFS data.

The data is being compressed before delivery, reducing object size.

Why wrong: Compression is optional.

The data transformation lambda is splitting records into smaller ones.

Why wrong: Not a typical behavior.

The S3 bucket is configured with a lifecycle policy that splits objects.

Why wrong: No such policy.

Use S3 Batch Operations to write data in batches.

Why wrong: Batch Operations is for post-hoc processing, not real-time streaming writes.

Increase the number of shards in the Kinesis data stream.

Why wrong: This addresses Kinesis throughput but not S3 throttling.

Enable S3 Transfer Acceleration on the destination bucket.

Why wrong: This improves transfer speed but does not handle throttling errors.

Write the aggregated results to a single large file instead of multiple partitions.

Why wrong: Single file reduces parallelism and increases shuffle overhead.

Convert the Parquet files to CSV to simplify the schema.

Why wrong: CSV is less efficient than Parquet for columnar storage and compression.

Replace the Redshift target with Amazon Redshift Spectrum.

Why wrong: Spectrum is for querying S3, not for loading transformed data into Redshift.

Amazon S3 Transfer Acceleration with a cron job.

Why wrong: Transfer Acceleration only accelerates uploads; it does not manage scheduling.

AWS Glue with a JDBC connection and a scheduled crawler.

Why wrong: Glue can do this but is more complex and less optimized for incremental replication.

Amazon Kinesis Data Firehose with a database source.

Why wrong: Firehose does not natively connect to relational databases.

Convert the CSV files to JSON format before processing.

Why wrong: JSON is less memory-efficient than CSV.

Decrease the Spark shuffle partitions in the job script.

Why wrong: This might reduce memory but is less direct and could impact performance.

Increase the job timeout setting.

Why wrong: Timeout only affects job duration, not memory.

Disable checkpointing to avoid failures.

Why wrong: Disabling checkpointing removes fault tolerance.

Switch the state backend from in-memory to RocksDB.

Why wrong: RocksDB helps manage large state but does not directly fix checkpoint failures.

Increase the parallelism of the application.

Why wrong: Higher parallelism can increase checkpoint overhead.

Set up an S3 event notification to trigger the Lambda function on object creation.

Why wrong: This triggers Lambda after data is written, not before.

Subscribe the Lambda function to the CloudWatch Logs log group for the Firehose stream.

Why wrong: This is for monitoring, not for data transformation.

Have the Lambda function write the transformed data directly to the S3 bucket.

Why wrong: This bypasses Firehose and is not the intended design.

Increase the Spark shuffle partitions to 500.

Why wrong: More partitions can increase overhead; optimal value depends on data.

Write the output as JSON instead of Parquet to avoid compression overhead.

Why wrong: JSON is larger and slower to read than Parquet.

Enable compression on the Kinesis stream.

Why wrong: Compression on Kinesis does not affect Glue memory.

Change the output format from Parquet to ORC.

Why wrong: ORC is similar to Parquet; format change does not fix OOM.

Reduce the streaming batch size in the Glue job configuration.

Why wrong: Reducing batch size can help but does not address the root cause of insufficient memory.

Amazon Kinesis Data Streams with AWS Lambda for transformation and Amazon S3.

Why wrong: Requires custom exactly-once logic; Lambda has no native Parquet conversion.

Amazon Simple Queue Service (SQS) with AWS Lambda for transformation and Amazon S3.

Why wrong: SQS does not natively convert to Parquet or partition data.

AWS Glue streaming jobs consuming from Amazon Kinesis Data Streams and writing to Amazon S3.

Why wrong: Glue streaming jobs are more complex and do not directly integrate with IoT Core.