Courseiva

CCNA Data Engineering Questions

75 of 350 questions · Page 2/5 · Data Engineering · Answers revealed

Practice these questionsDomain overviewAll questions
76
MCQhard

A data engineer is setting up a data lake on Amazon S3 for a large retail company. The data includes customer transactions, inventory, and web logs. The company wants to use AWS Glue for ETL and Amazon Athena for ad-hoc queries. The data is partitioned by year, month, day, and hour. The engineer notices that Athena queries are slow and often scan large amounts of data even when only a specific hour is needed. The engineer has already enabled partitioning and used columnar formats like Parquet. What additional step should the engineer take to optimize query performance and reduce data scanned?

A.Use a coarser partition layout, such as partitioning only by date, and leverage Hive-style partitioning with AWS Glue Crawlers to avoid excessive small files.
B.Convert the Parquet files to CSV format to reduce the overhead of columnar storage and improve compression.
C.Use S3 Select to push down filters to S3, reducing the amount of data scanned by Athena.
D.Increase the granularity of partitioning to include minute-level partitions to further limit data scanned.
AnswerA

Coarser partitions reduce the number of partitions and improve query planning.

Why this answer

Partitioning at a granularity of hour can result in a large number of small files, increasing metadata overhead and slowing query planning. By using a coarser partition layout (e.g., by date) and leveraging Hive-style partitioning with AWS Glue Crawlers, the number of partitions is reduced, which improves query performance and reduces the amount of data scanned. Option B is incorrect because converting Parquet to CSV would increase storage and scan costs due to lack of columnar compression and predicate pushdown.

Option C is incorrect because S3 Select operates on a single object, not across partitions; it is not designed for optimizing Athena queries over many files. Option D is incorrect because increasing partition granularity (e.g., minute-level) would create even more small files, worsening the issue.

Practice this question →
77
MCQmedium

A company uses Amazon Kinesis Data Streams for real-time clickstream analysis. The data is consumed by a Lambda function that enriches the records and stores them in Amazon S3. Recently, the Lambda function has been failing with throttling errors, and the consumer is falling behind. The team needs to increase the throughput of the consumer without changing the data format or the Lambda function code. What should the team do?

A.Add a second Kinesis data stream and send duplicate records to both.
B.Increase the batch size in the event source mapping for Lambda.
C.Increase the number of shards in the Kinesis data stream.
D.Increase the reserved concurrency of the Lambda function.
AnswerC

More shards increase the stream's capacity and number of Lambda consumers.

Why this answer

Increase the number of shards in the Kinesis data stream. Each shard supports a fixed number of read transactions per second and a maximum data read rate. Increasing the number of shards increases the parallelism of the stream, allowing the Lambda function to process records from multiple shards concurrently, thus increasing throughput.

Option A is incorrect because adding a second stream would require duplicating data and does not address the throttling on the existing stream. Option B is incorrect because increasing the batch size may reduce the number of Lambda invocations but does not increase the parallelism of the stream; the bottleneck is the shard count. Option D is incorrect because increasing reserved concurrency does not overcome the limitation that each shard can only trigger one Lambda invocation at a time; the main constraint is the number of shards.

Practice this question →
78
MCQmedium

A data scientist needs to run complex ETL transformations on a large dataset stored in Amazon S3. The transformations are written in PySpark and require occasional access to Hive metastore. The solution should minimize operational overhead and allow the data scientist to focus on code development. Which AWS service should be used?

A.Amazon Redshift
B.Amazon EMR
C.AWS Glue
D.Amazon SageMaker
AnswerB

EMR provides a managed Spark environment with Hive support and allows custom PySpark code.

Why this answer

Amazon EMR is the correct choice because it natively supports PySpark and Hive metastore integration, allowing the data scientist to run complex ETL transformations on large datasets stored in S3 with minimal operational overhead. EMR provides managed clusters that automatically scale and handle infrastructure, enabling the data scientist to focus on code development rather than cluster management.

Exam trap

The trap here is that candidates often confuse AWS Glue's serverless Spark environment with the ability to run arbitrary PySpark code with Hive metastore access, but Glue abstracts away cluster management and does not provide the same level of control or direct Hive metastore integration as EMR.

How to eliminate wrong answers

Option A is wrong because Amazon Redshift is a data warehouse optimized for SQL-based analytics and does not natively run PySpark transformations; it would require additional services or workarounds to execute PySpark code. Option C is wrong because AWS Glue is a serverless ETL service that uses Apache Spark under the hood but abstracts away cluster management and does not provide direct access to Hive metastore for occasional use; it is designed for simpler, automated ETL jobs rather than complex, custom PySpark transformations requiring metastore access. Option D is wrong because Amazon SageMaker is a machine learning platform focused on building, training, and deploying models, not for running general-purpose ETL transformations or PySpark jobs with Hive metastore access.

Practice this question →
79
MCQhard

A company uses Amazon Kinesis Data Analytics for real-time anomaly detection on a stream of IoT sensor data. The application is experiencing high latency. The data volume has doubled. Which action would MOST effectively reduce latency?

A.Increase the Parallelism setting of the Kinesis Data Analytics application
B.Change the record format from JSON to Avro
C.Decrease the retention period of the source stream
D.Increase the number of shards in the source Kinesis stream
AnswerA

More KPUs allow parallel processing of records.

Why this answer

Increasing the Parallelism setting of the Kinesis Data Analytics application directly allocates more processing resources (e.g., more Kinesis Processing Units or KPUs) to handle the doubled data volume. This allows the application to process records concurrently, reducing the per-record processing time and overall latency. Parallelism is the primary scaling mechanism for Kinesis Data Analytics to match throughput increases.

Exam trap

The trap here is that candidates often confuse scaling the source stream (shards) with scaling the processing application (parallelism), mistakenly thinking that increasing shards will automatically reduce latency, when in fact the bottleneck is the application's compute capacity, not the stream's ingestion rate.

How to eliminate wrong answers

Option B is wrong because changing the record format from JSON to Avro reduces data size and may improve deserialization efficiency, but it does not address the root cause of high latency from doubled data volume—it only optimizes the existing processing path without adding capacity. Option C is wrong because decreasing the retention period of the source Kinesis stream only controls how long data is stored before automatic deletion; it does not affect the rate at which data is consumed or processed by the analytics application, so it cannot reduce current processing latency. Option D is wrong because increasing the number of shards in the source Kinesis stream increases the ingestion capacity and read throughput, but the bottleneck is in the Kinesis Data Analytics application's processing capacity, not in data ingestion; without increasing the application's parallelism, the additional shards will not reduce latency and may even cause backpressure.

Practice this question →
80
Multi-Selecthard

A data engineer is designing a data pipeline to process streaming data from Amazon Kinesis Data Streams and store the results in Amazon S3 in Parquet format. The data must be available for querying in Amazon Athena within minutes of arrival. Which THREE services should be used together? (Choose THREE.)

Select 3 answers
A.Amazon EMR
B.Amazon Redshift
C.Amazon Kinesis Data Firehose
D.Amazon Kinesis Data Analytics
E.AWS Glue
AnswersC, D, E

Amazon Kinesis Data Firehose is correct because it is a fully managed service that can directly ingest streaming data from Kinesis Data Streams, convert it to Parquet format, and deliver it to Amazon S3 with minimal latency (typically 60 seconds). This enables near-real-time querying via Athena without custom code or infrastructure management.

Why this answer

Amazon Kinesis Data Firehose can directly ingest streaming data from Kinesis Data Streams, convert it to Parquet, and deliver to S3 with low latency. Amazon Kinesis Data Analytics can process and analyze the stream in real-time (e.g., aggregations or filtering) before sending to Firehose. AWS Glue provides a data catalog for the S3 data, making it queryable by Athena.

Together, these three services enable near-real-time querying of streaming data in Athena.

Exam trap

The trap is that candidates may overlook Kinesis Data Analytics if they think only Firehose and Glue are needed, but Data Analytics enables real-time processing transformations that are often required in machine learning pipelines. Alternatively, they might incorrectly include EMR or Redshift, which are not necessary for this simple streaming-to-S3 pattern.

Practice this question →
81
MCQhard

An IAM policy attached to an AWS Glue job allows reading and writing to an S3 bucket and accessing Glue Data Catalog. The job fails with an access denied error when trying to create a table in the Data Catalog. What is the likely issue?

A.The Glue Data Catalog is not enabled for the account.
B.The job does not have permission to write to the S3 bucket.
C.The S3 bucket is encrypted with a KMS key that the job cannot access.
D.The policy does not include the glue:CreateTable action.
AnswerD

Only GetTable and GetDatabase are allowed, not CreateTable.

Why this answer

The policy allows GetTable and GetDatabase actions, but not CreateTable. The job needs glue:CreateTable permission. The S3 actions are sufficient.

The error is specifically about creating a table.

Practice this question →
82
MCQmedium

A company is building a data pipeline that ingests data from multiple sources into a centralized data lake on Amazon S3. The data must be transformed before it is available for analysis. The pipeline should be event-driven, automatically triggering transformation jobs when new data arrives. Which combination of AWS services should be used?

A.Amazon Kinesis Data Analytics for transformation
B.Amazon S3 event notifications to invoke AWS Lambda, which triggers an AWS Glue job
C.Amazon EMR with automatic scaling
D.AWS Step Functions to orchestrate the pipeline
AnswerB

S3 events trigger Lambda, which starts a Glue ETL job; this is event-driven and serverless.

Why this answer

Amazon S3 event notifications can be configured to invoke an AWS Lambda function when new objects are created in an S3 bucket. The Lambda function can then trigger an AWS Glue job to perform the necessary data transformations. This creates an event-driven, serverless pipeline that automatically processes data as it arrives, meeting the requirements for a centralized data lake on S3.

Exam trap

The trap here is that candidates may choose AWS Step Functions (Option D) because it is a powerful orchestrator, but they overlook that it is not directly event-driven from S3 without an intermediary like Lambda or EventBridge, and it does not perform the actual transformation.

How to eliminate wrong answers

Option A is wrong because Amazon Kinesis Data Analytics is designed for real-time stream processing using SQL or Apache Flink, not for batch transformation jobs triggered by new data arriving in S3. Option C is wrong because Amazon EMR with automatic scaling is a managed Hadoop cluster for big data processing, but it is not inherently event-driven; it requires additional services like Lambda or Step Functions to trigger jobs based on S3 events, making it an overcomplicated and less direct solution. Option D is wrong because AWS Step Functions is a workflow orchestration service that can coordinate multiple AWS services, but it does not directly react to S3 events; it would need an event source like S3 notifications or EventBridge to start execution, and it is not a transformation service itself.

Practice this question →
83
Multi-Selecthard

A data engineer is designing an ETL pipeline using AWS Glue to process data from Amazon S3 and load it into Amazon Redshift. The pipeline must handle incremental data loads and ensure data consistency. Which THREE features should the engineer use to achieve this? (Choose THREE.)

Select 3 answers
A.Pushdown predicates to filter partitions in S3
B.Glue data preview to validate transformation logic
C.Glue partition filters to limit data scanned
D.Redshift transactional tables with automatic commit
E.Glue job bookmarks to track processed data
AnswersA, D, E

Pushdown predicates reduce the amount of data read from S3, improving performance.

Why this answer

(pushdown predicates) filters S3 partitions, reducing data scanned and enabling efficient incremental loads. Option D (Redshift transactional tables with automatic commit) ensures data consistency during writes. Option E (Glue job bookmarks) tracks processed data, supporting incremental processing.

Option B (Glue data preview) is used for development and does not contribute to incremental loading or consistency. Option C (Glue partition filters) is less efficient than pushdown predicates for filtering partitions.

Practice this question →
84
MCQmedium

A company uses AWS Glue to catalog data in S3. Data is partitioned by year, month, day. The Glue crawler runs daily but sometimes misses new partitions. What should be done to ensure all partitions are cataloged?

A.Use a custom classifier to detect partition patterns.
B.Increase the crawler schedule to run every hour.
C.Configure the crawler to update all partitions on each run.
D.Enable partition indexing in the Glue table properties.
AnswerC

Configuring the crawler to update all partitions on each run ensures that the crawler scans the entire dataset and registers any new or missed partitions.

Why this answer

Configuring the Glue crawler to 'update all partitions on each run' forces the crawler to scan the entire S3 path and register any new partitions it finds, even if the partition structure hasn't changed. This ensures all missed partitions are cataloged. Partition indexing (D) improves query performance by creating an index over existing partitions, but does not automatically discover new partitions.

Exam trap

The trap is that candidates may confuse partition indexing (which optimizes queries on already-cataloged partitions) with automatic partition discovery. The correct solution is to adjust the crawler's update behavior to scan all partitions, not to rely on indexing for cataloging.

How to eliminate wrong answers

Option A is wrong because custom classifiers are used to infer the schema of data formats (e.g., CSV, JSON) and do not affect partition discovery or cataloging. Option B is wrong because increasing the crawler schedule to run every hour does not guarantee that all partitions are cataloged if the crawler fails or if partitions are added between runs; it only reduces the window of missed partitions but does not solve the underlying issue of missed partitions. Option C is wrong because configuring the crawler to update all partitions on each run would be inefficient and does not address the root cause of missed partitions; the crawler still depends on its schedule and may skip partitions if they are not present during the crawl.

Practice this question →
85
Multi-Selectmedium

A data engineering team is designing a data lake on AWS for machine learning workloads. The data includes structured, semi-structured, and unstructured data. The team needs to ensure that the data is cataloged, easily discoverable, and can be queried by Amazon Athena and Amazon EMR. The team also wants to enforce fine-grained access control at the column and row level for sensitive data. Which combination of AWS services should the team use? (Select TWO.)

Select 2 answers
A.AWS Lake Formation
B.AWS Identity and Access Management (IAM)
C.AWS Glue Data Catalog
D.Amazon RDS for PostgreSQL
E.Amazon DynamoDB
AnswersA, C

Lake Formation provides fine-grained access control and integrates with Glue Catalog.

Why this answer

AWS Lake Formation is correct because it provides a centralized service to build, secure, and manage data lakes on AWS. It enables fine-grained access control at the column and row level for sensitive data, which directly meets the requirement for enforcing such controls. Additionally, Lake Formation integrates with Amazon Athena and Amazon EMR for querying and processing the cataloged data.

Exam trap

The trap here is that candidates often assume IAM alone can handle fine-grained data access control, but IAM lacks the column- and row-level filtering capabilities that Lake Formation provides through its integration with the Glue Data Catalog and query engines.

Practice this question →
86
Multi-Selectmedium

A company wants to use Amazon SageMaker to train a model using data stored in Amazon S3. The data is sensitive and must be encrypted at rest and in transit. Which THREE steps should be taken to ensure data security?

Select 3 answers
A.Configure the SageMaker training job to use an IAM role with least privilege and enable network isolation
B.Enable default encryption on the S3 bucket using AWS KMS
C.Use an S3 VPC endpoint to keep traffic within the AWS network
D.Store the data in Amazon Redshift instead of S3
E.Allow internet access for the SageMaker notebook instance
AnswersA, B, C

Network isolation ensures no internet egress.

Why this answer

Configuring the SageMaker training job with an IAM role that follows least privilege ensures that only necessary permissions are granted, reducing the risk of unauthorized access. Enabling network isolation prevents the training job from accessing the internet, which mitigates data exfiltration risks and ensures that data remains within the controlled AWS environment.

Exam trap

The trap here is that candidates might think storing data in a different service like Redshift or enabling internet access for notebooks is necessary, but the exam tests the understanding that S3 with VPC endpoints and network isolation provide sufficient security without overcomplicating the architecture.

Practice this question →
87
MCQeasy

A team is building a data pipeline using Amazon Kinesis Data Firehose to deliver real-time clickstream data to an Amazon S3 bucket. The data must be partitioned by year, month, day, and hour. Which configuration should the team use to achieve this?

A.Configure an S3 lifecycle rule to move data into partition folders after delivery
B.Use an AWS Lambda function to write data to S3 with the desired partition structure
C.Enable dynamic partitioning in Firehose and configure the partition keys as YYYY/MM/dd/HH
D.Use Amazon Athena partition projection to dynamically create partitions
AnswerC

Firehose dynamic partitioning automatically creates folder structures.

Why this answer

Amazon Kinesis Data Firehose supports dynamic partitioning, which allows you to automatically partition incoming data in S3 based on keys like YYYY/MM/dd/HH. By enabling this feature and configuring the partition keys to match the desired year, month, day, and hour format, Firehose will write data directly into the corresponding S3 prefix structure without requiring additional processing.

Exam trap

The trap here is that candidates often confuse S3 lifecycle rules or Athena partition projection as methods for creating partition structures, when in fact they are post-ingestion management or query-time features, not ingestion-time partitioning mechanisms.

How to eliminate wrong answers

Option A is wrong because S3 lifecycle rules are used for managing object lifecycle (e.g., transitioning to Glacier or deleting), not for creating partition folders at delivery time; they cannot retroactively reorganize data into a partition structure. Option B is wrong because while a Lambda function could write data with a custom partition structure, this approach adds complexity, latency, and cost, and is not the recommended or native way to achieve partitioning with Firehose; Firehose's built-in dynamic partitioning is designed for this exact use case. Option D is wrong because Athena partition projection is a feature for querying data in S3 by dynamically inferring partitions, not for writing or organizing data into partitioned folders during ingestion.

Practice this question →
88
MCQmedium

A data engineer runs the AWS CLI command above to inspect an object in S3. The engineer wants to query this metadata (kafka-offset) using Amazon Athena to track processing progress. How can the engineer make this metadata available for Athena queries without modifying the existing data pipeline?

A.Use S3 object tags instead of metadata and query the tags using Athena.
B.Use an AWS Lambda function to copy the metadata into the object's content as a new line.
C.Use AWS Glue to create a table that includes the metadata as a column by running an ETL job.
D.Use Amazon Athena to query the object metadata directly by referencing the metadata field.
AnswerC

A Glue ETL job can read objects, extract metadata, and write to a table that Athena can query.

Why this answer

AWS Glue ETL jobs can read the S3 object's user-defined metadata (e.g., 'kafka-offset') and write it as a column in a new or transformed dataset, which Athena can then query. This approach does not modify the existing data pipeline, as the original objects remain unchanged; the metadata is extracted and stored in a queryable format (e.g., Parquet or CSV) in a separate location. Glue's ability to access S3 object metadata via the `getObjectMetadata` API during ETL processing makes this a clean, pipeline-agnostic solution.

Exam trap

The trap here is that candidates assume Athena can natively query S3 object metadata (like HTTP headers) because Athena can query data in S3, but Athena has no access to object-level metadata—it only reads the content of files, not the object's key-value metadata fields.

How to eliminate wrong answers

Option A is wrong because S3 object tags are separate from user-defined metadata and cannot be directly queried by Athena; Athena queries data in files, not object tags, and there is no built-in Athena integration for tag-based queries. Option B is wrong because copying metadata into the object's content as a new line would modify the original object, violating the requirement to not alter the existing data pipeline, and it would also require additional orchestration to avoid race conditions or data duplication. Option D is wrong because Athena cannot query S3 object metadata directly; Athena only reads the content of objects (e.g., CSV, JSON, Parquet) and has no SQL access to HTTP headers or object-level metadata fields.

Practice this question →
89
MCQmedium

A team wants to build a data pipeline that processes incoming JSON files from an S3 bucket and loads them into a Redshift table. The pipeline must handle schema evolution and data validation. Which combination of services would be MOST appropriate?

A.Amazon S3 + AWS Glue + Amazon Redshift
B.Amazon S3 + Amazon SQS + Amazon Redshift
C.Amazon S3 + AWS Data Pipeline + Amazon Redshift
D.Amazon S3 + AWS Lambda + Amazon Redshift
AnswerA

Glue provides schema inference and ETL.

Why this answer

AWS Glue provides built-in schema discovery and evolution capabilities via its crawlers and the Data Catalog, which automatically detect and adapt to changes in JSON schemas. Combined with Glue ETL jobs for data validation and transformation, it seamlessly loads processed data into Amazon Redshift, making it the most appropriate choice for handling schema evolution and validation in this pipeline.

Exam trap

The trap here is that candidates often choose AWS Lambda for its simplicity and event-driven nature, overlooking its limitations with large files, lack of schema evolution, and inability to perform complex ETL within execution constraints.

How to eliminate wrong answers

Option B is wrong because Amazon SQS is a message queuing service that does not provide schema evolution, data validation, or ETL capabilities; it would only decouple components without addressing the core requirements. Option C is wrong because AWS Data Pipeline is a batch-oriented orchestration service that lacks native schema discovery and evolution features, requiring manual handling of schema changes and validation logic. Option D is wrong because AWS Lambda is stateless and has a 15-minute execution timeout, making it unsuitable for processing large JSON files or complex ETL tasks, and it does not offer built-in schema evolution or data validation capabilities.

Practice this question →
90
Multi-Selecthard

A company is using Amazon DynamoDB as a source for a machine learning pipeline. The data is exported nightly to Amazon S3 using DynamoDB Streams and an AWS Glue job. The Glue job reads the stream records, transforms them, and writes to S3 in Parquet format. The team notices that the Glue job is taking too long and consuming high DynamoDB read capacity. Which THREE actions would reduce the load on DynamoDB and improve performance? (Choose THREE.)

Select 3 answers
A.Use Amazon DynamoDB export to S3 (incremental) feature instead of Glue
B.Increase the DynamoDB write capacity units to handle the stream writes
C.Use DynamoDB Streams with AWS Lambda to write data directly to S3 in near-real-time, bypassing Glue
D.Increase the DynamoDB read capacity units to handle Glue's workload
E.Configure Glue to read from a S3 snapshot exported earlier instead of directly from DynamoDB
AnswersA, C, E

The export feature does not consume read capacity and can be automated.

Why this answer

DynamoDB's native export to S3 (incremental) feature directly exports data to S3 without consuming read capacity units (RCUs) or requiring a separate compute service like AWS Glue. This eliminates the bottleneck of Glue reading from DynamoDB Streams, which consumes RCUs and adds latency, thereby reducing load on DynamoDB and improving overall performance.

Exam trap

The trap here is that candidates often assume increasing DynamoDB capacity (RCUs or WCUs) is the solution to performance issues, but the exam tests understanding that native export features and architectural changes (like using Lambda or S3 snapshots) can eliminate the root cause of high read consumption without scaling capacity.

Practice this question →
91
Multi-Selectmedium

Which THREE factors should a data engineer consider when choosing between Amazon S3 and Amazon Redshift for storing large datasets used for machine learning? (Choose 3.)

Select 3 answers
A.Query performance and latency requirements
B.Encryption at rest capabilities
C.Cost of storage vs. compute
D.Data format and compression support
E.Data retention policies
AnswersA, C, D

Redshift provides fast SQL analytics; S3 queries are slower.

Why this answer

When choosing between Amazon S3 and Amazon Redshift for ML data storage, key considerations include: (A) Query performance and latency: Redshift offers low-latency SQL querying on structured data, while S3 provides higher latency for direct access, making performance needs critical. (C) Cost of storage vs. compute: S3 decouples storage and compute, allowing independent scaling; Redshift combines them, affecting cost. (D) Data format and compression: S3 supports any format, but Redshift works best with columnar formats like Parquet. (B) Encryption at rest and (E) Data retention policies are available in both, so they are not differentiating factors.

Exam trap

A common mistake is assuming encryption or retention policies are unique to one service, when in fact both S3 and Redshift offer equivalent capabilities, making them irrelevant for this comparison.

Practice this question →
92
Multi-Selectmedium

A company is using Amazon Kinesis Data Streams to ingest clickstream data. The data is consumed by a fleet of EC2 instances running a custom consumer application. The consumer is falling behind and the shard iterator age is increasing. Which TWO actions should the data engineer take to improve consumer performance? (Choose TWO.)

Select 2 answers
A.Increase the number of shards in the stream
B.Decrease the data retention period
C.Use an AWS Lambda function to process the data
D.Enable enhanced fan-out on the stream
E.Switch to the Kinesis Client Library (KCL)
AnswersA, D

More shards increase the total read capacity.

Why this answer

Increasing the number of shards in the stream directly increases the total read capacity of the Kinesis Data Stream. Each shard provides a fixed read throughput of 2 MB/s (or 5 read transactions per second), so adding shards allows the consumer fleet to parallelize processing across more data partitions, reducing the backlog and shard iterator age.

Exam trap

The trap here is that candidates often confuse 'switching to KCL' (a library) with a performance fix, when in fact KCL is just a helper for checkpointing and load balancing, not a throughput booster.

Practice this question →
93
MCQmedium

A data engineer runs the AWS CLI command shown in the exhibit to find large log files in S3. The command returns an empty list, but the engineer knows there are files larger than 1 MB in that prefix. What is the MOST likely issue?

A.The JMESPath query syntax is incorrect
B.The command does not paginate through all objects; only the first 1000 are returned
C.The prefix is incorrect; there are no objects under that prefix
D.The Size value is in kilobytes, not bytes
AnswerB

list-objects limits to 1000 keys; use --max-items or pagination.

Why this answer

The `list-objects` command returns up to 1000 objects per call. If there are more than 1000 objects under the prefix, the command only examines the first 1000 objects. Since the engineer knows there are files larger than 1 MB, those files likely appear after the first 1000 objects.

To find them, pagination is required (e.g., using `--page-size` or `--max-items` and `--starting-token`). Option A is incorrect because the JMESPath query is syntactically valid. Option C is incorrect because the engineer confirmed objects exist under the prefix.

Option D is incorrect because `Size` is in bytes, so `1000000` correctly represents 1 MB.

Practice this question →
94
MCQmedium

A company captures streaming data from IoT devices using Amazon Kinesis Data Streams. The data is consumed by a custom application that processes records in near real-time. Recently, the application has been falling behind, and the stream is showing increased 'iterator age' metrics in CloudWatch. Which action is MOST likely to reduce the iterator age?

A.Increase the data retention period of the stream
B.Decrease the number of shards in the stream
C.Increase the number of shards in the stream
D.Reduce the data retention period of the stream
AnswerC

More shards increase throughput, allowing the consumer to process faster.

Why this answer

The 'iterator age' metric in Amazon Kinesis Data Streams measures the time between the oldest unread record in a shard and the current time. An increasing iterator age indicates that consumers are reading data slower than it is being produced. Increasing the number of shards increases the stream's total read capacity, allowing the custom application to process records in parallel and reduce the backlog.

Exam trap

Common misconception: increasing retention period helps with processing backlogs, but retention only affects data durability, not throughput; the correct solution is to scale shards to match consumer throughput.

How to eliminate wrong answers

Option A is wrong because increasing the data retention period (up to 365 days) only extends how long records are stored, not the throughput capacity; it does not help consumers catch up. Option B is wrong because decreasing the number of shards reduces the stream's read and write capacity, worsening the backlog and increasing iterator age. Option D is wrong because reducing the data retention period (minimum 24 hours) would cause older unprocessed records to be deleted, but it does not increase read throughput or help the application process faster.

Practice this question →
95
Multi-Selecthard

A company uses Amazon Redshift for data warehousing. The data engineering team notices that query performance has degraded over time. Which THREE actions should the team take to improve performance? (Choose THREE.)

Select 3 answers
A.Increase the number of nodes in the Redshift cluster
B.Define appropriate sort keys on large tables
C.Define appropriate distribution keys on large tables
D.Delete old data that is no longer needed
E.Run the ANALYZE command to update table statistics
AnswersB, C, E

Sort keys minimize the amount of data scanned, improving query performance.

Why this answer

Sort keys in Amazon Redshift define the order in which data is stored on disk within each node. By defining appropriate sort keys on large tables, the query engine can use zone maps to skip entire blocks of data that do not match the query's filter predicates, dramatically reducing the amount of data scanned and improving query performance.

Exam trap

The trap here is that candidates often confuse scaling up (adding nodes) with performance tuning, but the MLS-C01 exam expects you to recognize that data engineering best practices—like proper sort keys, distribution keys, and updated statistics—are the primary levers for improving query performance in Redshift, not just adding more hardware.

Practice this question →
96
MCQmedium

A data scientist is using Amazon SageMaker to train a model. The training data is stored in Amazon S3 and is approximately 500 GB. The data scientist notices that the training job is taking a long time to start because the data is being copied to the training instance's storage. The data scientist wants to reduce the startup time for subsequent training jobs. Which action should the data scientist take?

A.Use Pipe input mode instead of File input mode for the training job
B.Use an EBS-optimized instance type
C.Use Amazon FSx for Lustre as a high-performance file system mounted to the training instance
D.Increase the size of the training instance's Amazon EBS storage volume
AnswerA

Pipe mode streams data from S3 directly, reducing startup time.

Why this answer

Using Pipe input mode streams data directly from S3 to the training algorithm without downloading, reducing startup time. Option B is wrong because FSx for Lustre is not needed for simple streaming. Option C is wrong because increasing instance storage does not address the data transfer issue.

Option D is wrong because using EBS optimized instances does not change the data loading mechanism.

Practice this question →
97
MCQmedium

A company is using Amazon Kinesis Data Firehose to deliver streaming data to an S3 bucket. The data is JSON and must be transformed into Parquet format before delivery. Which approach should the data engineer use?

A.Send the data to Amazon Kinesis Data Analytics to convert to Parquet
B.Configure Kinesis Data Firehose to convert the record format to Parquet using a schema from AWS Glue Data Catalog
C.Use an AWS Lambda function to transform JSON to Parquet and write to S3
D.Use an AWS Glue ETL job to read from Firehose and write Parquet to S3
AnswerB

Firehose can convert JSON to Parquet using a Glue Data Catalog schema.

Why this answer

Amazon Kinesis Data Firehose can directly convert incoming JSON records to Parquet format by referencing a schema stored in the AWS Glue Data Catalog. This is a built-in feature of Firehose that does not require additional services for the conversion. Option A is wrong because Kinesis Data Analytics is for real-time analytics, not format conversion.

Option C is wrong because while Lambda can transform data, using it for Parquet conversion adds latency and complexity; Firehose's native conversion is simpler. Option D is wrong because an AWS Glue ETL job is for batch processing, not real-time streaming transformation.

Practice this question →
98
MCQeasy

A company is using Amazon S3 as a data lake. The data engineering team needs to catalog the schema of the data and make it available for querying with Amazon Athena. Which AWS Glue component should be used?

A.AWS Glue Studio
B.AWS Glue Crawlers
C.AWS Glue ETL jobs
D.AWS Glue DataBrew
AnswerB

Crawlers populate the Glue Data Catalog with table definitions.

Why this answer

AWS Glue Crawlers automatically scan data in Amazon S3, infer the schema, and populate the AWS Glue Data Catalog with metadata tables. This makes the data immediately available for querying with Amazon Athena without manual schema definition.

Exam trap

The trap here is that candidates confuse Glue Crawlers (schema discovery) with Glue ETL jobs (data transformation) or Glue DataBrew (data preparation), assuming any Glue component can catalog data, but only Crawlers perform automatic schema inference and metadata population.

How to eliminate wrong answers

Option A is wrong because AWS Glue Studio is a visual interface for authoring ETL jobs, not for cataloging schemas. Option C is wrong because AWS Glue ETL jobs are used for transforming and moving data, not for schema discovery and cataloging. Option D is wrong because AWS Glue DataBrew is a visual data preparation tool for cleaning and normalizing data, not for automatic schema inference and catalog population.

Practice this question →
99
Multi-Selecthard

A company uses Amazon S3 to store historical transaction data in CSV format. The data is partitioned by transaction_date. A data analyst runs Amazon Athena queries that frequently filter on customer_id and transaction_date. The queries are slow and expensive. The team needs to improve query performance and reduce cost. Which combination of actions should the team take? (Choose TWO.)

Select 2 answers
A.Enable S3 Select pushdown in Athena to reduce data transfer.
B.Convert the data to JSON format for better query performance.
C.Convert the data from CSV to Parquet format.
D.Reorganize the data by partitioning on customer_id first, then transaction_date.
E.Increase the number of Athena query workers.
AnswersC, D

Parquet is columnar and compressed, reducing data scanned.

Why this answer

Converting CSV to Parquet (option C) improves performance because Parquet is a columnar storage format that reduces the amount of data scanned by Athena, especially when queries only select a subset of columns. It also uses efficient compression, reducing storage and data scanned. Reorganizing the partition order (option D) to have customer_id first (the most frequently filtered column) improves partition pruning, reducing the amount of data read.

Option A (S3 Select pushdown) is not fully supported by Athena; Athena already uses S3 Select for certain formats, but it doesn't guarantee significant improvement and may not be applicable. Option B (JSON) is worse than CSV because JSON is typically larger and not columnar. Option E (increasing workers) is not applicable as Athena is serverless and automatically scales.

Practice this question →
100
MCQmedium

A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Lambda function that writes to an S3 bucket. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors. What is the MOST likely cause?

A.The data retention period of the stream is too short.
B.The S3 bucket has insufficient write capacity.
C.The Kinesis stream has too few shards for the data volume.
D.The Lambda function's reserved concurrency is set too high.
AnswerC

Insufficient shards cause ProvisionedThroughputExceededException.

Why this answer

The 'ProvisionedThroughputExceededException' error in Amazon Kinesis Data Streams indicates that the data ingestion rate exceeds the write capacity of the stream's shards. Each shard supports up to 1 MB/s or 1,000 records/s for writes. If the clickstream data volume surpasses this limit, the Lambda function, which reads from the stream, will encounter this exception.

Increasing the number of shards scales the write capacity to match the data volume.

Exam trap

The trap here is that candidates confuse Kinesis throughput limits with Lambda concurrency or S3 capacity, but the specific exception name 'ProvisionedThroughputExceededException' is a direct indicator of insufficient shard write capacity in Kinesis.

How to eliminate wrong answers

Option A is wrong because the data retention period (default 24 hours, up to 365 days) controls how long records are stored, not the write throughput; a short retention period would cause data loss, not throughput errors. Option B is wrong because S3 buckets have virtually unlimited write capacity (thousands of PUT requests per second per prefix) and do not produce 'ProvisionedThroughputExceededException' errors, which are specific to Kinesis. Option D is wrong because setting reserved concurrency too high for the Lambda function would not cause a Kinesis throughput error; it might lead to throttling of the Lambda itself, but the exception originates from the Kinesis stream's shard limits.

Practice this question →
101
MCQeasy

An AWS Glue job is failing with an error that it cannot access an S3 bucket. The IAM role attached to the Glue job is shown in the exhibit. What is the MOST likely cause of the failure?

A.The S3 bucket has a bucket policy that denies access to this role
B.The role lacks S3 permissions
C.The role does not have permission to call S3 APIs
D.The trust policy does not allow Glue to assume the role
AnswerA

A bucket policy can override the role's permissions.

Why this answer

Even if the IAM role has S3 permissions, an S3 bucket policy that explicitly denies access to that role will override any allow. AWS evaluates all policies (identity-based and resource-based) and a deny in any policy results in a final deny decision. The error indicates the Glue job cannot access the bucket, which is consistent with a bucket-level deny.

Exam trap

The trap here is that candidates assume the IAM role's permissions are the only factor, ignoring that S3 bucket policies can independently deny access, which overrides any allow in the role's policy.

How to eliminate wrong answers

Option B is wrong because the IAM role shown in the exhibit likely includes S3 permissions (e.g., s3:GetObject, s3:PutObject) — the question states the role is attached, so lacking S3 permissions is not the most likely cause given the error. Option C is wrong because 'lacks S3 permissions' and 'does not have permission to call S3 APIs' are essentially the same misconception; the role may have API permissions but be blocked by the bucket policy. Option D is wrong because if the trust policy did not allow Glue to assume the role, the job would fail with an assume-role error, not an S3 access error.

Practice this question →
102
MCQmedium

A data engineering team is building a real-time fraud detection system. Transactions are ingested via Amazon Kinesis Data Streams, and a machine learning model (deployed on Amazon SageMaker) scores each transaction. The team needs to store the raw transactions and the model's predictions in Amazon S3 for later analysis. Which architecture should the team use?

A.Use AWS Lambda to read from Kinesis, invoke SageMaker, and write directly to S3.
B.Use Amazon Kinesis Data Firehose with a transformation Lambda to call SageMaker.
C.Use Amazon Kinesis Data Analytics for Apache Flink to enrich records with SageMaker predictions, then output to Firehose for S3.
D.Use AWS Lambda to invoke the SageMaker endpoint for each record, then write to S3 via Firehose.
AnswerC

Flink can handle high-throughput, call SageMaker per record, and output to Firehose.

Why this answer

It uses Amazon Kinesis Data Analytics for Apache Flink to perform real-time enrichment by invoking the SageMaker endpoint for each transaction, then streams the enriched records to Kinesis Data Firehose for reliable, batched delivery to S3. This architecture handles the asynchronous nature of model inference without blocking the ingestion stream, and Firehose provides automatic retry and compression for S3 storage.

Exam trap

The trap here is that candidates often assume Lambda is the only serverless option for real-time enrichment, but the exam tests whether you understand that Kinesis Data Analytics for Apache Flink is the correct service for asynchronous, stateful enrichment before delivery to S3 via Firehose.

How to eliminate wrong answers

Option A is wrong because AWS Lambda reading directly from Kinesis and writing to S3 would require per-record Lambda invocations, leading to high latency, potential throttling, and no built-in buffering or retry mechanism for S3 writes. Option B is wrong because Kinesis Data Firehose's transformation Lambda is synchronous and cannot asynchronously invoke an external endpoint like SageMaker; it is designed for simple record transformations, not for making external HTTP calls that may time out or fail. Option D is wrong because using Lambda to invoke SageMaker and then write to S3 via Firehose adds unnecessary complexity and latency, as Firehose expects a stream of records, not individual Lambda outputs; this approach also duplicates the buffering logic that Firehose already provides.

Practice this question →
103
MCQhard

A company is using Amazon Kinesis Data Analytics for Apache Flink to process real-time sensor data. The application reads from a Kinesis data stream, performs windowed aggregations, and writes results to an S3 bucket. Recently, the application has been experiencing high latency and checkpoint failures. What is the MOST likely cause?

A.The number of shards in the Kinesis stream is insufficient for the data volume
B.The S3 destination bucket is located in a different AWS Region than the Kinesis application
C.The record size in the Kinesis stream exceeds the 1 MB limit
D.The parallelism of the Flink application is set too low for the number of shards
AnswerB

Cross-region writes increase latency and can cause checkpoint timeouts.

Why this answer

The S3 destination bucket is located in a different AWS Region than the Kinesis application. Cross-region data transfer introduces network latency and increases the likelihood of checkpoint failures because Apache Flink checkpoints require writes to complete within a timeout. Option A (insufficient shards) would cause throttling (ProvisionedThroughputExceededException) but not directly checkpoint failures.

Option C (record size > 1 MB) is impossible because Kinesis Data Streams enforces a 1 MB maximum record size. Option D (low parallelism) could cause backpressure but not typically checkpoint failures unless resources are severely constrained. Therefore, the most likely cause is the cross-region S3 bucket.

Practice this question →
104
Multi-Selecthard

Which THREE of the following are best practices for optimizing performance of Amazon EMR clusters? (Choose 3)

Select 3 answers
A.Use Spot Instances for task nodes
B.Consolidate small files into larger ones before processing
C.Use instance fleets for heterogeneous instances
D.Enable EBS optimization on EC2 instances
E.Use Spot Instances to reduce costs
AnswersB, C, D

Consolidation reduces overhead and improves performance.

Why this answer

Consolidating small files into larger ones before processing on Amazon EMR reduces the overhead of the Hadoop Distributed File System (HDFS) metadata operations. Each small file consumes a block of memory in the NameNode, and processing many small files leads to excessive task launches and I/O overhead, degrading performance. Using tools like `s3-dist-cp` to combine files into fewer, larger blocks improves throughput and reduces job execution time.

Exam trap

The trap here is that candidates confuse cost optimization strategies (like Spot Instances) with performance optimization, leading them to select options A or E even though the question explicitly asks for performance best practices.

Practice this question →
105
MCQhard

Refer to the exhibit. A data engineer runs the AWS CLI command to check an object in an S3 bucket. The bucket is part of a data lake and is configured with versioning enabled. However, the output shows "VersionId": null. What is the most likely reason for this?

A.The object is encrypted using SSE-S3, which hides the version ID
B.The object was uploaded before versioning was enabled
C.The command must include the --version-id parameter to display the version ID
D.Versioning is not enabled on the bucket
AnswerB

Objects uploaded before versioning was enabled have a null version ID.

Why this answer

The most likely reason is that the object was uploaded before versioning was enabled on the bucket. When versioning is enabled, objects uploaded afterward receive a unique version ID, while objects that existed before versioning was enabled have a null version ID. Option B is correct.

Option A is incorrect because SSE-S3 encryption does not affect version IDs; version IDs are metadata independent of encryption. Option C is incorrect because the `head-object` command automatically returns the version ID of the latest version; no `--version-id` parameter is needed to display it. Option D is incorrect because the bucket is explicitly stated to have versioning enabled.

Practice this question →
106
MCQhard

A company uses Amazon EMR to run Spark jobs on a transient cluster that processes data from S3. The jobs are failing with 'OutOfMemory' errors. The data engineer has already increased the executor memory. Which additional configuration change would MOST likely resolve the issue?

A.Use fewer, larger instance types for the core nodes
B.Increase the number of partitions in the data
C.Increase the driver memory
D.Increase the number of executors
AnswerB

More partitions means smaller data per task, reducing memory usage.

Why this answer

The 'OutOfMemory' errors in Spark on EMR typically occur when individual partitions hold too much data for the executor's memory to process. Increasing the number of partitions distributes the data more evenly across available memory, reducing the per-partition size and preventing memory overflow during shuffle or aggregation operations. This directly addresses the root cause of memory pressure, whereas simply increasing executor memory may only delay the failure.

Exam trap

The trap here is that candidates often assume adding more memory (executor or driver) or scaling vertically (larger instances) is the solution, but the exam tests understanding that memory errors in Spark are frequently caused by partition size imbalance, not insufficient total memory.

How to eliminate wrong answers

Option A is wrong because using fewer, larger instance types reduces the total number of cores and task slots, which can actually increase memory pressure per executor and worsen OutOfMemory errors. Option C is wrong because driver memory is used for the Spark driver process (e.g., collecting results, scheduling), not for executor-side data processing; increasing it does not help with executor OutOfMemory errors. Option D is wrong because increasing the number of executors without adjusting partitions can lead to more tasks competing for the same data, but each executor still processes the same large partitions, so memory errors persist.

Practice this question →
107
MCQmedium

A team is building a data pipeline that ingests data from an Amazon S3 bucket, transforms it using AWS Glue, and loads it into Amazon Redshift for analysis. The Glue job runs on a schedule every hour. The team has noticed that the job takes longer than expected and sometimes fails due to memory issues. The data volume is variable, with occasional spikes. Which solution should the team implement to optimize the pipeline?

A.Decrease the number of workers to reduce memory contention.
B.Enable job bookmarks to process only new data and use a G.2X worker type for more memory.
C.Increase the schedule frequency to run the job more often with smaller data increments.
D.Replace AWS Glue with Amazon EMR using Spark.
AnswerB

Job bookmarks prevent reprocessing and larger workers provide more memory.

Why this answer

Enabling job bookmarks allows the Glue job to process only new or changed data since the last run, reducing the data volume per execution. Using the G.2X worker type provides additional memory (e.g., 16 GB per DPU vs. 4 GB for G.1X), which helps prevent out-of-memory failures during data spikes. Together, these optimizations address both the variable data volume and memory constraints without requiring a complete pipeline redesign.

Exam trap

The trap here is that candidates may assume increasing job frequency (Option C) will automatically reduce per-run data volume, but without incremental processing (job bookmarks), each run still processes the entire dataset, leading to the same memory issues and higher costs.

How to eliminate wrong answers

Option A is wrong because decreasing the number of workers reduces parallelism and available memory, which would likely worsen performance and increase the chance of memory failures. Option C is wrong because increasing the schedule frequency does not reduce the per-run data volume unless combined with incremental processing; it would only run the same full dataset more often, potentially increasing resource contention and cost. Option D is wrong because replacing AWS Glue with Amazon EMR is an unnecessary architectural change; Glue is already suitable for this use case, and the issues can be resolved with proper configuration (job bookmarks and worker type) without migrating to a more complex managed Spark cluster.

Practice this question →
108
MCQeasy

A data scientist uses Amazon SageMaker to train a model. The training dataset is 10 GB and stored in S3. The training job uses a ml.m5.large instance. The data must be available on the local file system during training. Which input mode should be used?

A.Local input mode
B.Batch input mode
C.File input mode
D.Pipe input mode
AnswerC

File mode downloads data to the local file system, making it available for training.

Why this answer

File input mode is correct because it downloads the entire training dataset from S3 to the local file system of the ml.m5.large instance before training begins, ensuring the data is available locally as required. This mode is suitable for datasets up to 10 GB, as the instance's local storage (typically 8 GB for ml.m5.large) may be insufficient, but SageMaker uses the instance's Amazon EBS volume (up to 512 GB) for file input mode, making it viable.

Exam trap

The trap here is that candidates may confuse 'File input mode' with 'Pipe input mode' and incorrectly choose Pipe mode for local file availability, or invent 'Local input mode' as a plausible-sounding option.

How to eliminate wrong answers

Option A is wrong because 'Local input mode' is not a valid SageMaker input mode; the correct term is 'File input mode' for local file system access. Option B is wrong because 'Batch input mode' is not a SageMaker input mode; SageMaker uses 'File' or 'Pipe' modes, and batch processing refers to Batch Transform jobs, not training input. Option D is wrong because 'Pipe input mode' streams data directly from S3 to the training algorithm without writing to the local file system, which does not satisfy the requirement that data must be available on the local file system during training.

Practice this question →
109
MCQeasy

A data engineering team needs to process streaming data from thousands of IoT devices. The data must be ingested with low latency and processed in near real-time to detect anomalies. Which AWS service should they use for ingestion?

A.Amazon Kinesis Data Firehose
B.Amazon Kinesis Data Analytics
C.Amazon S3
D.Amazon Kinesis Data Streams
AnswerD

Kinesis Data Streams is the correct service for real-time streaming ingestion.

Why this answer

Amazon Kinesis Data Streams is the correct choice because it is designed for real-time, low-latency ingestion of streaming data from thousands of sources, such as IoT devices. It provides a durable, scalable data stream that can be consumed by multiple applications in near real-time, making it ideal for anomaly detection use cases.

Exam trap

The trap here is that candidates confuse Kinesis Data Firehose (which delivers data to destinations with some latency) with Kinesis Data Streams (which is designed for real-time ingestion and processing), often overlooking the 'low latency' and 'near real-time' requirements in the question.

How to eliminate wrong answers

Option A is wrong because Amazon Kinesis Data Firehose is a near-real-time delivery service that buffers data before writing it to destinations like S3 or Redshift, introducing latency of up to 60 seconds, which is not suitable for low-latency ingestion. Option B is wrong because Amazon Kinesis Data Analytics is a service for processing and analyzing streaming data using SQL or Apache Flink, not for ingestion itself. Option C is wrong because Amazon S3 is an object storage service with eventual consistency and higher latency for writes, making it unsuitable for real-time streaming ingestion from thousands of IoT devices.

Practice this question →
110
MCQhard

A data pipeline uses AWS Lambda to process small files (10-50 MB) from an S3 bucket and write results to DynamoDB. The Lambda function times out after 15 seconds for larger files. The team wants to handle files up to 100 MB without changing the Lambda code. Which solution is MOST cost-effective?

A.Use AWS Glue Python shell job to replace Lambda
B.Increase the Lambda function timeout to 5 minutes
C.Use Amazon ECS with AWS Fargate to run the processing task
D.Configure an SQS queue to buffer the S3 events and batch them
AnswerB

Lambda allows up to 15 minutes, and 5 minutes is sufficient for 100 MB. No code changes needed.

Why this answer

Increasing the Lambda function timeout to 5 minutes directly addresses the 15-second timeout issue for larger files (up to 100 MB) without requiring any code changes. This is the most cost-effective solution as it avoids additional infrastructure costs (e.g., Glue, ECS, SQS) and leverages Lambda's existing pay-per-execution pricing model, which remains economical for occasional longer-running invocations.

Exam trap

The trap here is that candidates assume Lambda is unsuitable for larger files or longer processing times, leading them to over-engineer with services like Glue or ECS, when simply increasing the timeout is the most cost-effective and minimal-change solution.

How to eliminate wrong answers

Option A is wrong because replacing Lambda with an AWS Glue Python shell job introduces unnecessary complexity and cost (Glue charges per DPU-hour) for a simple file processing task that Lambda can handle with a timeout adjustment. Option C is wrong because using Amazon ECS with AWS Fargate adds operational overhead and cost (per vCPU and memory) for a task that Lambda can perform more simply and cheaply with a timeout increase. Option D is wrong because configuring an SQS queue to buffer S3 events and batch them does not solve the Lambda timeout issue; batching would still require each Lambda invocation to process a file within the timeout, and it adds latency and complexity without addressing the root cause.

Practice this question →
111
MCQmedium

A media company ingests video metadata from multiple sources into an Amazon S3 bucket. Each metadata record is a JSON file about 2 KB. They use AWS Glue ETL jobs to process these files and load them into Amazon Redshift for analytics. The jobs currently run hourly and take about 10 minutes to process all new files. However, the company is growing and expects the number of files to increase 100x. The data engineering team wants to minimize processing time and cost. The Glue job currently reads all files from the S3 bucket using a full scan. What should they do to optimize the pipeline?

A.Consolidate the small JSON files into larger files using a scheduled job
B.Convert the data to Parquet format and partition it
C.Increase the number of Glue DPUs to process files faster
D.Use S3 event notifications to trigger Glue jobs only for new files
AnswerD

S3 event notifications allow the Glue job to be triggered for only new objects, so only new files are processed, eliminating unnecessary full scans.

Why this answer

Using S3 event notifications to trigger Glue jobs only for new files eliminates the need to scan all files in the bucket, reducing processing time and cost. Option A consolidating files would reduce the number of small files but does not address the full scan issue and would still require processing all files. Option B converting to Parquet improves performance and reduces scan size, but the job still scans all files unnecessarily.

Option C increasing DPUs speeds up processing but increases cost without addressing the root cause of scanning all files.

Practice this question →
112
MCQmedium

A company is building a data pipeline to process sensitive customer data. The pipeline uses AWS Glue for ETL and stores results in Amazon S3. The security team requires that all data be encrypted at rest in S3 using customer-managed AWS KMS keys. Additionally, the Glue job must be able to write encrypted data to S3. What should the data engineer do to meet these requirements?

A.Attach a policy to the Glue job's IAM role that includes kms:GenerateDataKey and kms:Decrypt actions for the KMS key.
B.Use S3 server-side encryption with customer-provided keys (SSE-C).
C.Use S3 server-side encryption with SSE-S3, which is enabled by default.
D.Configure an S3 bucket policy to enforce encryption and attach it to the Glue job's IAM role.
AnswerA

These permissions allow Glue to encrypt and decrypt data using the KMS key.

Why this answer

AWS Glue jobs use an IAM role to interact with AWS services. To write encrypted data to S3 using a customer-managed AWS KMS key, the IAM role must have permissions for `kms:GenerateDataKey` (to request a data key for encryption) and `kms:Decrypt` (to decrypt the data key when reading or writing). This allows the Glue job to encrypt objects at rest in S3 with the specified KMS key, meeting the security team's requirement.

Exam trap

The trap here is that candidates often assume an S3 bucket policy alone can enforce encryption without realizing that the IAM role performing the write must also have explicit KMS permissions for the customer-managed key.

How to eliminate wrong answers

Option B is wrong because SSE-C requires the customer to provide the encryption key in each request, which is not suitable for automated Glue jobs and does not use AWS KMS keys. Option C is wrong because SSE-S3 uses AWS-managed keys, not customer-managed KMS keys, failing the requirement for customer-controlled encryption. Option D is wrong because an S3 bucket policy can enforce encryption (e.g., deny unencrypted writes), but it does not grant the Glue job's IAM role the necessary KMS permissions to encrypt data; the IAM role must still have explicit KMS actions allowed.

Practice this question →
113
MCQhard

A company is building a near-real-time dashboard using data from multiple sources. They need to aggregate millions of events per second with sub-second latency. The architecture must be fully managed and minimize operational overhead. Which service should they use for the aggregation layer?

A.Amazon Kinesis Data Analytics for Apache Flink.
B.AWS Lambda functions triggered by Kinesis Data Streams.
C.Amazon EMR with Spark Streaming.
D.Amazon Redshift with materialized views refreshed frequently.
AnswerA

Kinesis Data Analytics with Flink provides low-latency, stateful stream processing at scale.

Why this answer

Amazon Kinesis Data Analytics for Apache Flink is the correct choice because it provides a fully managed, serverless runtime for Apache Flink, which is designed for stateful stream processing at scale. It can aggregate millions of events per second with sub-second latency using exactly-once semantics and built-in checkpointing, meeting the near-real-time dashboard requirements without any infrastructure management.

Exam trap

The trap here is that candidates often confuse AWS Lambda's event-driven nature with true stream processing, overlooking its concurrency and latency limitations for high-throughput aggregation, or they assume Spark Streaming is always the best choice for real-time without considering Flink's superior sub-second latency and fully managed nature on Kinesis Data Analytics.

How to eliminate wrong answers

Option B is wrong because AWS Lambda functions triggered by Kinesis Data Streams have a maximum concurrency limit and a 15-minute execution timeout, making them unsuitable for aggregating millions of events per second with sub-second latency; Lambda is better for lightweight, stateless transformations. Option C is wrong because Amazon EMR with Spark Streaming requires manual cluster provisioning, scaling, and maintenance, increasing operational overhead, and Spark Streaming typically has higher latency (seconds) compared to Flink's sub-second capabilities. Option D is wrong because Amazon Redshift with materialized views refreshed frequently is designed for batch-oriented, analytical queries on structured data, not for real-time stream aggregation; it cannot handle millions of events per second with sub-second latency and introduces significant refresh overhead.

Practice this question →
114
MCQhard

An e-commerce company uses Amazon Kinesis Data Firehose to deliver clickstream data to Amazon S3. The data arrives at unpredictable rates, with occasional bursts. The company needs to ensure data is delivered within 60 seconds of ingestion, and the data must be partitioned by year/month/day/hour. Which configuration meets these requirements?

A.Set the buffer size to 1 MB and disable dynamic partitioning
B.Use a Lambda function to process data and write to S3 with partitioning
C.Use AWS Glue streaming ETL to read from Firehose and write to S3
D.Set the buffer interval to 60 seconds and enable dynamic partitioning
AnswerD

Buffer interval controls delivery frequency; dynamic partitioning creates time-based folders.

Why this answer

Setting the buffer interval to 60 seconds ensures data is flushed to Amazon S3 within that time window, meeting the 60-second delivery requirement. Enabling dynamic partitioning allows Firehose to automatically partition data by year/month/day/hour based on the data's timestamp, without needing custom code or additional services.

Exam trap

The trap here is that candidates may think a Lambda function or Glue ETL is required for custom partitioning, when Firehose's native dynamic partitioning can handle time-based partitioning directly with a simple configuration change.

How to eliminate wrong answers

Option A is wrong because setting the buffer size to 1 MB and disabling dynamic partitioning does not guarantee delivery within 60 seconds (buffer interval defaults to 300 seconds) and cannot partition data by year/month/day/hour. Option B is wrong because using a Lambda function to process data and write to S3 with partitioning introduces additional complexity, latency, and potential for data loss or duplication, and is not a native Firehose configuration. Option C is wrong because AWS Glue streaming ETL reads from Kinesis Data Streams, not directly from Firehose, and adds unnecessary overhead and cost for a simple partitioning and delivery requirement.

Practice this question →
115
MCQhard

A data engineering team is designing a data lake on Amazon S3. They need to enforce encryption at rest for all data stored in the bucket. The security policy requires that the encryption keys be managed by the organization using AWS Key Management Service (KMS), and that the bucket must deny uploads of unencrypted objects. Which bucket policy should be applied?

A.A bucket policy that denies PutObject unless the request includes the 'x-amz-server-side-encryption' header with value 'AES256'
B.A bucket policy that denies PutObject if the 'x-amz-server-side-encryption' header is not present
C.A bucket policy that denies PutObject unless the request includes the 'x-amz-server-side-encryption-aws-kms-key-id' header matching the desired KMS key ID
D.Enable default encryption on the bucket with AWS-KMS
AnswerC

This enforces the use of a specific KMS key.

Why this answer

The security policy requires that encryption keys be managed by the organization using AWS KMS, and that unencrypted uploads be denied. A bucket policy that denies PutObject unless the request includes the 'x-amz-server-side-encryption-aws-kms-key-id' header matching the desired KMS key ID enforces both conditions: it ensures server-side encryption with a customer-managed KMS key (SSE-KMS) and blocks any upload that does not specify the exact key ID, thereby preventing unencrypted objects or objects encrypted with other keys.

Exam trap

The trap here is that candidates often confuse 'default encryption' (which silently encrypts objects but does not deny unencrypted uploads) with a bucket policy that actively denies requests without the required encryption headers, leading them to choose Option D instead of the policy-based enforcement in Option C.

How to eliminate wrong answers

Option A is wrong because it enforces SSE-S3 (AES256), not SSE-KMS, which violates the requirement that encryption keys be managed by the organization via AWS KMS. Option B is wrong because it only checks for the presence of the 'x-amz-server-side-encryption' header but does not enforce that the encryption uses a KMS key; the header could be set to 'AES256' (SSE-S3) or 'aws:kms' (SSE-KMS), and without specifying the key ID, it does not meet the key management requirement. Option D is wrong because enabling default encryption on the bucket does not deny uploads of unencrypted objects; it only applies encryption to objects that are uploaded without an encryption header, meaning a client could still upload an unencrypted object if they explicitly set the header to 'None' or omit it, and the bucket would still accept it (default encryption is a fallback, not a denial).

Practice this question →
116
MCQeasy

A machine learning team is using Amazon SageMaker to train models on a large dataset stored in Amazon S3. The dataset is 5 TB in size and is partitioned by date. The team wants to minimize data transfer costs and reduce training time by caching frequently accessed data locally on the training instances. The training instances are EC2 instances with attached Amazon EBS volumes. The team is considering using SageMaker Pipe mode to stream data directly from S3, but they are concerned about network bandwidth. Which approach should the team use to optimize data loading for training?

A.Use Amazon FSx for Lustre as a high-performance file system linked to the S3 bucket, and mount it on the training instances.
B.Use SageMaker File mode with Amazon EFS, which allows multiple training instances to share the same file system and caches data from S3.
C.Increase the size of the EBS volumes attached to the training instances and copy the entire dataset to the volumes before training.
D.Use SageMaker Pipe mode to stream data from S3 directly to the training algorithm, which automatically caches data in memory.
AnswerA

Correct. Amazon FSx for Lustre provides a high-performance file system that integrates with S3 and caches data locally on the training instances, reducing data transfer costs and training time.

Why this answer

Amazon FSx for Lustre is natively integrated with Amazon SageMaker as a data source, providing a high-performance file system that can be linked directly to an S3 bucket. It automatically caches frequently accessed data from S3 on the file system, reducing data transfer costs and training time by avoiding repeated downloads. The caching capability addresses network bandwidth concerns effectively.

Option B is incorrect: SageMaker File mode uses EBS volumes, not Amazon EFS, and is not designed as a shared, cached file system across training jobs. Option C is incorrect: copying the entire 5 TB dataset to EBS volumes before each training job is time-consuming, increases costs, and does not provide efficient caching across jobs. Option D is incorrect: SageMaker Pipe mode streams data directly from S3 without caching, so it does not reduce repeated data transfers and may still face bandwidth issues.

Practice this question →
117
MCQeasy

A data engineer needs to extract data from an Amazon RDS for MySQL database into Amazon S3 for further processing. The data volume is 2 TB and the job must run daily within a 1-hour window. Which AWS service is most suitable for this task?

A.Amazon Kinesis Data Firehose
B.AWS Database Migration Service (DMS)
C.Amazon Athena
D.AWS Glue
AnswerD

AWS Glue provides managed ETL jobs that can extract from JDBC sources and write to S3 on a schedule.

Why this answer

AWS Glue is the most suitable service because it provides a fully managed ETL (Extract, Transform, Load) capability that can efficiently extract 2 TB of data from Amazon RDS for MySQL and write it to Amazon S3. Glue can leverage JDBC connections to the RDS instance, scale horizontally with its dynamic worker allocation, and complete the job within a 1-hour window by using appropriate worker types (e.g., G.2X or G.8X) and partitioning strategies. Additionally, Glue integrates natively with the AWS Glue Data Catalog and can handle incremental or full-load extraction with minimal overhead.

Exam trap

The trap here is that candidates often confuse AWS Glue (a batch ETL service) with Amazon Kinesis Data Firehose (a streaming service) or AWS DMS (a migration tool), failing to recognize that Glue's Spark-based parallel processing and JDBC connectivity make it the correct choice for scheduled, large-volume batch extraction from a relational database to S3.

How to eliminate wrong answers

Option A is wrong because Amazon Kinesis Data Firehose is designed for streaming data ingestion (e.g., real-time logs, events) and does not support batch extraction from a relational database like RDS for MySQL; it lacks JDBC connectors and cannot perform scheduled bulk reads. Option B is wrong because AWS Database Migration Service (DMS) is primarily intended for one-time or ongoing database migrations (e.g., to another database engine or S3), but it is not optimized for daily, time-boxed ETL jobs with a strict 1-hour window; DMS can incur latency from change data capture and may require additional configuration for partitioning large datasets. Option C is wrong because Amazon Athena is an interactive query service that runs SQL directly on data in S3; it cannot extract data from an external database like RDS for MySQL—it has no built-in JDBC connectivity to pull data from RDS.

Practice this question →
118
Drag & Dropmedium

Drag and drop the steps to create an Amazon SageMaker notebook instance in the correct order.

Drag steps to the numbered slots on the right, or tap a step then tap a slot.

Steps
Order
1Step 1
2Step 2
3Step 3
4Step 4

Why this order

Creating a notebook instance requires navigating the SageMaker console, configuring instance settings, IAM role, and VPC, then launching.

Practice this question →
119
MCQhard

A data engineer is investigating why an Athena query against the my-data-lake bucket is slow. The query filters on year, month, and day. The exhibit shows the metadata of one Parquet file. What is the MOST likely cause of the slow query?

A.The version ID is null, causing data inconsistency
B.The file is too large, causing Athena to process it in a single task
C.The partition columns are not being used in the query
D.The storage class is STANDARD, which is slower than GLACIER
AnswerB

Large files limit parallelism; Athena works best with files 128-512 MB.

Why this answer

The Parquet file is 1 GB in size, which is too large for efficient processing in Athena. Athena splits data into tasks for parallel execution, but a single large file cannot be split, causing the query to run slowly. Partitioning on year, month, and day is already applied and is not the issue.

The other options are incorrect: version ID null is irrelevant, the query does use partition columns, and standard storage is faster than Glacier.

Practice this question →
120
MCQhard

A company is streaming data from thousands of devices using Amazon Kinesis Data Streams. The data is consumed by a AWS Lambda function that processes each record. The Lambda function is experiencing high error rates and throttling due to the volume of data. Which action would MOST effectively improve the processing throughput and reduce errors?

A.Send the data to Amazon SQS first and then process with Lambda
B.Use Amazon Kinesis Data Firehose instead of Kinesis Data Streams
C.Increase the Lambda function's batch size and reduce the batch window
D.Increase the number of shards in the Kinesis stream
AnswerD

More shards increase parallelism and throughput, reducing throttling.

Why this answer

Increasing the number of shards in the Kinesis stream directly increases the stream's capacity for data ingestion and processing parallelism. Each shard supports up to 1 MB/s or 1,000 records/s for writes, and Lambda processes records from each shard concurrently. By adding more shards, you distribute the load across more Lambda invocations, reducing throttling and error rates caused by exceeding the per-shard throughput limits.

Exam trap

The trap here is that candidates often confuse Kinesis Data Streams with Kinesis Data Firehose, thinking Firehose can handle high-volume Lambda processing, when in fact Firehose is a delivery service with no per-record Lambda integration.

How to eliminate wrong answers

Option A is wrong because inserting an SQS queue between Kinesis and Lambda adds an unnecessary hop and does not address the root cause of throttling; Lambda still polls SQS at a fixed rate, and the bottleneck remains the downstream processing capacity. Option B is wrong because Kinesis Data Firehose is designed for near-real-time data delivery to destinations like S3 or Redshift, not for real-time processing with Lambda; it does not support per-record Lambda processing and would not reduce errors from high-volume streaming. Option C is wrong because increasing the batch size while reducing the batch window can actually increase the number of records per invocation, potentially worsening throttling and error rates if the Lambda function cannot handle larger batches within the execution timeout or memory limits.

Practice this question →
121
Multi-Selecthard

Which THREE considerations are important when designing a data lake on Amazon S3?

Select 3 answers
A.Setting up S3 Lifecycle policies to transition data to colder storage
B.Using Provisioned IOPS for S3
C.Partitioning data by date to improve query performance
D.Using a single Availability Zone for data storage
E.Encrypting data at rest using AWS KMS
AnswersA, C, E

Lifecycle policies manage cost.

Why this answer

S3 Lifecycle policies allow you to automatically transition objects to colder storage classes (e.g., S3 Standard-IA, S3 Glacier) based on age or other rules, reducing storage costs for infrequently accessed data in a data lake. This is a key design consideration for managing data lifecycle and cost efficiency at scale.

Exam trap

The trap here is that candidates may confuse S3 with EBS features (like Provisioned IOPS) or overlook that S3 inherently provides multi-AZ durability, making single-AZ storage an anti-pattern for data lakes.

Practice this question →
122
MCQmedium

Refer to the exhibit. An ML engineer runs the above CLI command to inspect files in an S3 bucket. The training data consists of 200 CSV files, each 1 GB. The engineer plans to use Amazon SageMaker to train a model using this data. What should the engineer do to optimize training performance?

A.Increase the number of training instances to process files in parallel.
B.Use Amazon Athena to transform the data into CSV format with headers.
C.Use the File input mode and copy all files to the training instance's EBS volume.
D.Convert the CSV files to Parquet format and use Pipe input mode.
AnswerD

Parquet is columnar and compressed; Pipe mode streams data directly from S3.

Why this answer

Converting CSV files to Parquet format and using Pipe input mode significantly improves training performance. Parquet is a columnar storage format that reduces I/O by reading only relevant columns, and it is compressed. Pipe input mode streams data directly from S3 to the training algorithm without downloading to EBS, reducing startup time and disk usage.

Option A is incorrect because simply increasing the number of instances does not address the inefficiency of reading CSV files; it may help parallelization but not per-instance throughput. Option B is incorrect because Amazon Athena is a query service, not a data transformation tool for SageMaker; converting to CSV with headers does not improve performance. Option C is incorrect because using File input mode copies all files to the training instance's EBS volume, which is slow for 200 GB of data and does not leverage streaming benefits.

Practice this question →
123
MCQmedium

A data engineer is designing a data lake on Amazon S3. The data is collected from IoT devices and is highly variable in volume. The engineer needs to ensure that the data is ingested reliably and can be processed in near real-time. Which AWS service should be used to ingest the data into the data lake?

A.Amazon Kinesis Data Firehose
B.AWS Glue
C.Amazon Kinesis Data Streams
D.Amazon Simple Queue Service (SQS)
AnswerA

Firehose can load streaming data directly into S3 with near real-time latency.

Why this answer

Amazon Kinesis Data Firehose is the correct choice because it is a fully managed service designed to reliably load streaming data into data lakes on Amazon S3 with near-real-time latency (typically 60 seconds). It automatically handles scaling to accommodate highly variable IoT data volumes, provides built-in data transformation and compression, and requires no manual shard management or consumer code, making it ideal for ingestion into S3-based data lakes.

Exam trap

The trap here is that candidates often confuse Amazon Kinesis Data Streams (a raw streaming service requiring custom consumers) with Amazon Kinesis Data Firehose (a fully managed delivery service), leading them to select Data Streams for direct S3 ingestion when it actually requires additional code and infrastructure to write to S3.

How to eliminate wrong answers

Option B (AWS Glue) is wrong because it is a serverless ETL and data catalog service for batch processing and schema discovery, not a real-time data ingestion service; it cannot ingest streaming data directly into S3. Option C (Amazon Kinesis Data Streams) is wrong because it is a real-time data streaming service that requires you to write custom consumer code to read and write data to S3, and it does not natively deliver data to S3 without additional infrastructure; it is designed for custom stream processing, not direct ingestion into a data lake. Option D (Amazon Simple Queue Service (SQS)) is wrong because it is a message queue service for decoupling application components, not a streaming ingestion service; it does not provide near-real-time delivery to S3 and lacks built-in data transformation or compression capabilities for data lake ingestion.

Practice this question →
124
MCQhard

A data engineer configures an S3 event notification to trigger an AWS Lambda function when a new object is created in 'my-input-bucket'. The Lambda function processes the CSV file and writes results to 'my-output-bucket'. The engineer notices that the Lambda function is not triggered for some objects. Which step should the engineer take to diagnose the issue?

A.Check the Lambda function's execution role for permissions to write to the output bucket.
B.Review the CloudWatch Logs for the Lambda function to see if there are errors.
C.Check the Lambda function's resource-based policy to ensure S3 has permission to invoke the function.
D.Verify that the S3 event notification is configured with the correct prefix and suffix filters.
AnswerC

Missing invoke permission is a common cause of trigger failure.

Why this answer

The most likely cause of the Lambda function not being triggered for some objects is that S3 lacks the necessary permission to invoke the function. S3 event notifications require a resource-based policy (also known as a Lambda function policy) that explicitly grants the S3 service principal permission to invoke the function. Without this policy, S3 will not be able to trigger the Lambda function, even if the event notification configuration is correct.

Exam trap

The trap here is that candidates often focus on the Lambda function's execution role (Option A) or the event notification filters (Option D), overlooking the critical resource-based policy that grants S3 permission to invoke the function.

How to eliminate wrong answers

Option A is wrong because the Lambda function's execution role permissions to write to the output bucket affect the function's ability to write results, not whether the function is triggered in the first place. Option B is wrong because reviewing CloudWatch Logs would only help after the function has been invoked; if the function is never triggered, there will be no logs to review. Option D is wrong because while prefix and suffix filters could cause some objects to be excluded, the question states that the function is not triggered for 'some objects' — if the filters were the issue, the function would not be triggered for objects that do not match the filter, which is expected behavior, not a problem to diagnose.

Practice this question →
125
MCQhard

A data engineer runs the AWS CLI command shown and notices a zero-byte file in the results. What is the most likely cause of this zero-byte file?

A.The S3 bucket has a lifecycle policy that deleted the content.
B.The file was written with the wrong prefix.
C.The file was compressed, reducing size to zero.
D.The file was created by a failed Spark task that wrote no data.
AnswerD

Failed tasks can produce empty files.

Why this answer

Zero-byte files often occur when an ETL job fails partway through writing, or when a task starts but writes no data. A completed write would have non-zero size. The other options are less likely: prefix typo wouldn't produce a file; correct permissions wouldn't cause zero bytes; compression would produce some output.

Practice this question →
126
MCQhard

A large e-commerce company is using Amazon DynamoDB as the source for real-time analytics. The data is streamed to Amazon Kinesis Data Streams using DynamoDB Streams and then processed by an AWS Lambda function. The Lambda function writes the data to an Amazon Elasticsearch Service cluster for search and visualization. Recently, the Lambda function has been failing with throttling errors from the Elasticsearch cluster. What is the MOST effective way to handle this?

A.Increase the Lambda function's reserved concurrency to handle more invocations.
B.Increase the number of shards in the Kinesis data stream.
C.Decrease the Kinesis stream's retention period to reduce the data volume.
D.Configure a Dead Letter Queue (DLQ) on the Lambda function to capture failed records and implement retry logic.
AnswerD

DLQ captures records that fail due to throttling, allowing later reprocessing without blocking the stream.

Why this answer

Using a Dead Letter Queue (DLQ) allows the Lambda function to capture records that fail due to Elasticsearch throttling, so they can be retried later without blocking the function's processing of other records. This prevents data loss and handles backpressure effectively. Option A is incorrect because increasing Lambda concurrency would increase the rate of writes to the Elasticsearch cluster, worsening the throttling.

Option B is incorrect because increasing Kinesis shards would increase the throughput of data arriving at the Lambda function, again exacerbating the throttling. Option C is incorrect because decreasing the Kinesis retention period does not reduce the data volume; it only changes how long data is stored in the stream, and it would not solve the throttling issue.

Practice this question →
127
MCQhard

A company is using Amazon Kinesis Data Analytics for Apache Flink to process real-time data. The data source is a Kinesis data stream, and the output is written to an S3 bucket. Recently, the processing latency has increased significantly. The team suspects that the Flink application is encountering backpressure. Which metric should the team monitor to confirm backpressure?

A.currentLowWatermark
B.busyTimeMsPerSecond
C.numberOfFailedCheckpoints
D.numRecordsInPerSecond
AnswerB

High busy time indicates operator is overloaded, causing backpressure.

Why this answer

The correct metric to confirm backpressure in a Flink application is `busyTimeMsPerSecond`. This metric measures the percentage of time a task is actively processing data versus waiting for input. A high `busyTimeMsPerSecond` value (close to 1000ms) indicates that the task is fully utilized and cannot keep up with the incoming data rate, which is the direct symptom of backpressure.

Other metrics like `currentLowWatermark` relate to event time progress, not backpressure.

Exam trap

The trap here is that candidates often confuse `currentLowWatermark` (event time progress) with backpressure detection, or they assume that a high input rate (`numRecordsInPerSecond`) automatically means backpressure, but backpressure is about the operator's inability to keep up, not just the volume of data.

How to eliminate wrong answers

Option A is wrong because `currentLowWatermark` tracks the progress of event time processing and is used for watermark alignment and out-of-order event handling, not for detecting backpressure. Option C is wrong because `numberOfFailedCheckpoints` indicates checkpoint failures, which can be a consequence of backpressure but are not a direct measure of backpressure itself; they could also result from other issues like state size or network failures. Option D is wrong because `numRecordsInPerSecond` shows the input rate but does not indicate whether the operator is struggling to process that rate; a high input rate alone does not confirm backpressure.

Practice this question →
128
MCQmedium

The Glue job my-glue-job fails after a few successful runs. The error log shows 'Job run exceeds max concurrent runs limit'. The CloudFormation template is shown in the exhibit. What change should be made to allow multiple runs to execute concurrently?

A.Change the IAM role to one with more permissions
B.Increase the MaxRetries property to 3
C.Remove the --job-bookmark-option argument
D.Set the MaxConcurrentRuns property to 3
AnswerD

This allows up to 3 concurrent job runs.

Why this answer

The 'MaxConcurrentRuns' is set to 1, which prevents parallel executions. Setting it to a higher value (e.g., 3) allows concurrent runs. MaxRetries is for retry count, not concurrency.

Role and TempDir are not relevant.

Practice this question →
129
MCQmedium

A data science team uses Amazon SageMaker to train models on a large dataset stored in S3. The dataset is 500 GB in CSV format and is updated daily. The team wants to optimize data loading for training jobs to reduce I/O wait time. Which data ingestion strategy is MOST effective?

A.Use SageMaker File input mode and increase the EBS volume size to 1 TB.
B.Use SageMaker Pipe input mode to stream data directly from S3.
C.Convert the CSV files to Parquet format and use File input mode.
D.Load the data into an Amazon EFS file system and mount it to the training instance.
AnswerB

Pipe mode streams data on-the-fly, eliminating the need to download the full dataset, thus reducing I/O wait time.

Why this answer

SageMaker Pipe input mode streams data directly from S3 to the training algorithm without writing to the instance's EBS volume, eliminating disk I/O bottlenecks. This is especially effective for large datasets (500 GB) that are updated daily, as it reduces startup time and avoids the need to download the entire dataset before training begins.

Exam trap

The trap here is that candidates often assume converting to a columnar format like Parquet always improves performance, but they overlook that File input mode still requires a full download to disk, whereas Pipe mode avoids that entirely regardless of file format.

How to eliminate wrong answers

Option A is wrong because increasing the EBS volume size to 1 TB does not reduce I/O wait time; it only provides more storage space, and the data must still be downloaded from S3 to the EBS volume before training, which adds latency. Option C is wrong because while converting CSV to Parquet can improve read performance and reduce data size, using File input mode still requires the entire dataset to be downloaded to the instance's EBS volume before training starts, negating the benefit of reduced I/O wait time. Option D is wrong because mounting an Amazon EFS file system to the training instance introduces network file system latency and is not optimized for the high-throughput, low-latency data loading required for training jobs; SageMaker's built-in Pipe mode is designed specifically for this purpose.

Practice this question →
130
MCQeasy

A company uses AWS Lambda to process events from Amazon S3. The Lambda function transforms the data and writes results to another S3 bucket. Recently, the function has been failing due to timeout errors when processing large files. Which solution should the data engineer implement?

A.Increase the Lambda function memory and timeout limit
B.Increase the Lambda timeout to 15 minutes
C.Use S3 Batch Operations with a Lambda function to process objects
D.Use Amazon SQS to queue the events and process them in batches
AnswerC

Batch Operations can invoke Lambda for each object, handling large volumes.

Why this answer

S3 Batch Operations is designed to handle large-scale object processing by invoking a Lambda function asynchronously for each object, bypassing the synchronous invocation limits of S3 event notifications. This allows processing of large files without hitting Lambda's 15-minute timeout or memory constraints, as each object is processed independently and the operation can scale to billions of objects.

Exam trap

The trap here is that candidates assume increasing timeout or memory (Option A) is the universal fix for Lambda failures, but the real issue is the synchronous invocation model from S3 events, which S3 Batch Operations solves by decoupling the processing.

How to eliminate wrong answers

Option A is wrong because increasing memory and timeout only addresses symptoms of a single invocation limit, not the root cause of large file processing failures; Lambda's maximum timeout is 15 minutes regardless of memory. Option B is wrong because increasing timeout to 15 minutes is the maximum possible, but large files may still exceed this limit or cause memory exhaustion, and it does not solve the underlying issue of synchronous invocation constraints from S3 events. Option D is wrong because Amazon SQS queues events but does not change the per-invocation timeout or memory limits; batching events in SQS still results in individual Lambda invocations that can timeout on large files.

Practice this question →
131
MCQhard

A company needs to process sensitive data from multiple sources. They want to use AWS Glue to catalog and transform the data. Which feature should they use to ensure that sensitive columns are masked before the data is available for querying?

A.AWS Glue DataBrew
B.AWS Glue Studio
C.AWS Lake Formation
D.Amazon Macie
AnswerA

DataBrew allows data masking and cleansing interactively.

Why this answer

Glue DataBrew provides data masking and cleansing capabilities. Glue Studio is for building ETL jobs, but masking requires custom code. Lake Formation is for fine-grained access control, not masking.

Macie is for discovering sensitive data, not masking.

Practice this question →
132
MCQeasy

A data scientist needs to train a machine learning model using a large dataset (500 GB) stored in an S3 bucket. The training will be performed on a SageMaker notebook instance. The data scientist wants to minimize data transfer costs and reduce training time. Which data ingestion approach should the data engineer recommend?

A.Use the SageMaker SDK to directly read the data from S3 during training without copying it to the notebook.
B.Copy the dataset to the notebook instance's attached EBS volume before training.
C.Load the dataset into an Amazon RDS database and query it from the notebook.
D.Mount the S3 bucket to the notebook instance using Amazon Elastic File System (EFS).
AnswerA

SageMaker can read data directly from S3, minimizing transfer and storage costs.

Why this answer

The SageMaker SDK allows training jobs to read data directly from S3 using the Pipe or File mode, which avoids copying the 500 GB dataset to the notebook instance's EBS volume. This minimizes data transfer costs (no egress from S3 to the notebook) and reduces training time by streaming data directly to the training container without intermediate storage.

Exam trap

The trap here is that candidates often assume they must copy data locally for faster access (Option B), not realizing that SageMaker's native S3 integration with Pipe mode is designed specifically to avoid that overhead and is the most cost-effective and performant approach for large datasets.

How to eliminate wrong answers

Option B is wrong because copying the entire 500 GB dataset to the notebook instance's EBS volume incurs high data transfer costs from S3 to the instance and consumes significant time for the copy operation, plus the EBS volume may be too small or require additional provisioning. Option C is wrong because loading 500 GB into Amazon RDS introduces unnecessary complexity, higher costs for database storage and I/O, and querying over the network adds latency, which is inefficient for large-scale ML training. Option D is wrong because mounting S3 via Amazon EFS is not a supported or practical approach; EFS is a separate NFS-based file system, not a direct S3 mount, and would require additional services like EFS File Sync or FUSE, adding cost and complexity without the native streaming benefits of SageMaker's S3 integration.

Practice this question →
133
MCQhard

A data engineer created an IAM policy to allow a Glue ETL job to read and write objects to an S3 bucket. The ETL job fails when writing data with the error 'Access Denied'. The job is configured to use SSE-S3 (AES256) encryption. What is the likely issue?

A.The policy grants s3:PutObject on all buckets, not just the specific one.
B.The condition requires objects to be encrypted with SSE-KMS, but the job uses SSE-S3.
C.The policy does not grant s3:PutObject on the bucket itself, which is needed for some write operations.
D.The condition requires objects to use SSE-S3, but the job uses SSE-KMS.
AnswerC

Bucket-level permissions may be required for certain write operations.

Why this answer

The error 'Access Denied' when writing to S3 with SSE-S3 encryption typically occurs because the IAM policy lacks the `s3:PutObject` permission on the bucket resource itself. While the policy may grant `s3:PutObject` on the object ARN (`arn:aws:s3:::bucket/*`), some S3 write operations—especially those involving encryption headers or bucket-level checks—also require the permission on the bucket ARN (`arn:aws:s3:::bucket`). Without this, the request is denied even if the object-level permission exists.

Exam trap

The trap here is that candidates assume `s3:PutObject` on the object ARN is sufficient for all write operations, overlooking that S3 requires the same permission on the bucket ARN for certain encryption-related or bucket-policy-evaluation scenarios.

How to eliminate wrong answers

Option A is wrong because granting `s3:PutObject` on all buckets would be overly permissive, not restrictive; the issue is missing permission on the specific bucket, not an overly broad scope. Option B is wrong because the job uses SSE-S3, and the condition requiring SSE-KMS would cause a different error (e.g., 'The request was denied because the encryption key is not authorized'), not a generic 'Access Denied'. Option D is wrong because the job uses SSE-S3, not SSE-KMS, so a condition requiring SSE-S3 would actually match and not cause a denial.

Practice this question →
134
MCQmedium

A data engineering team is using Apache Spark on Amazon EMR to process streaming data from Amazon Kinesis Data Streams. The Spark application uses structured streaming to read from Kinesis, perform transformations, and write to Amazon S3 in Parquet format. The team notices that the application is falling behind and the processing latency is increasing. The Kinesis stream has 5 shards, and the EMR cluster has 5 core nodes of type r5.xlarge. The Spark application is configured with 5 executors, each with 2 cores and 8 GB memory. The team wants to reduce processing latency. Which change would be most effective?

A.Increase the executor memory to 16 GB.
B.Increase the number of shards in the Kinesis stream to 10 and increase the number of core nodes to 10.
C.Use a larger instance type for the core nodes, such as r5.4xlarge.
D.Change the output format from Parquet to CSV to reduce write time.
AnswerB

More shards increase parallelism, and more nodes allow more concurrent processing.

Why this answer

The number of shards (5) matches the number of executors (5), but each shard can be processed by a single executor. To increase parallelism, the team should increase the number of shards in the Kinesis stream and correspondingly increase the number of executors or cores. Alternatively, they can increase the number of cores per executor to allow parallel processing of multiple shards per executor.

Practice this question →
135
MCQhard

A company runs a data pipeline using AWS Glue ETL jobs that process about 10 TB of data daily from Amazon S3. The jobs are triggered by a schedule and write results to a separate S3 bucket. Recently, the jobs have been taking longer to complete, and the data engineering team has observed that the number of files in the source bucket has increased significantly, from thousands to millions of small files (each about 100 KB). The Glue jobs are configured to use the 'Group Files' option, but performance is still poor. The team needs to improve the job performance without changing the source data generation process. Which course of action should the team take?

A.Increase the number of DPUs allocated to the existing Glue job
B.Switch the ETL processing to Amazon EMR with Spark
C.Use AWS Lambda to pre-process the files and combine them
D.Create a separate Glue job that runs before the main job to consolidate small files into larger ones in the source bucket
AnswerD

Consolidation reduces the number of files, improving read performance.

Why this answer

The main performance bottleneck is the large number of small files, which causes high overhead in reading metadata and opening files. Option D addresses this by creating a separate Glue job that consolidates small files into larger files (e.g., 100 MB) before the main ETL job runs, reducing the file count and improving read performance. Option A is incorrect because increasing DPUs may provide more parallelism but does not solve the underlying small-file problem; the overhead of opening millions of files remains.

Option B is incorrect because switching to Amazon EMR with Spark would still encounter the same small-file issue unless additional measures (like coalesce or file compaction) are taken, which Option D already provides. Option C is incorrect because AWS Lambda has limitations on execution duration and memory, making it impractical to pre-process millions of small files efficiently.

Practice this question →
136
Multi-Selecthard

A data engineer is designing a data pipeline that uses Amazon Kinesis Data Streams to ingest real-time transaction data. The data must be processed in near real-time and stored in Amazon S3 for long-term analytics. The engineer wants to ensure data durability and exactly-once processing semantics. Which TWO actions should the engineer take? (Choose two.)

Select 2 answers
A.Use the Kinesis Producer Library (KPL) with exactly-once delivery.
B.Use AWS Glue streaming ETL with checkpointing.
C.Enable exactly-once delivery on Kinesis Data Firehose.
D.Use AWS Lambda with the Kinesis trigger and enable event source mapping with RetryAttempts set to 0.
E.Use Amazon SQS as the event source for downstream processing.
AnswersB, D

AWS Glue streaming ETL with checkpointing can achieve exactly-once processing by tracking progress and writing to a transactional data lake, making this a correct action.

Why this answer

Correct options: B and D. AWS Glue streaming ETL with checkpointing provides exactly-once processing semantics when writing to S3 using a transactional format like Delta Lake. Setting RetryAttempts to 0 on a Lambda event source mapping ensures that each record is processed only once (no retries), which avoids duplicate processing, though failures may cause data loss.

Options A and C do not guarantee exactly-once: KPL provides at-least-once with deduplication, and Kinesis Data Firehose provides at-least-once delivery to S3. Option E (Amazon SQS) is not part of the Kinesis pipeline and does not ensure exactly-once semantics.

Exam trap

Candidates often assume Kinesis Data Firehose provides exactly-once delivery to S3, but it actually provides at-least-once. Also, the Kinesis Producer Library (KPL) provides at-least-once with deduplication, not exactly-once.

Practice this question →
137
MCQeasy

A data scientist wants to explore a large dataset stored in Amazon S3 using SQL queries without moving the data. The dataset is in CSV format and is updated daily with new partitions. Which AWS service should be used to directly query the data in S3?

A.Amazon Athena
B.Amazon Redshift Spectrum
C.Amazon EMR
D.AWS Glue
AnswerA

Athena is purpose-built for querying data in S3 with no infrastructure to manage.

Why this answer

Amazon Athena is a serverless interactive query service that allows you to analyze data directly in Amazon S3 using standard SQL, without needing to load or transform the data. It supports CSV format and can automatically discover new partitions when used with Hive-style partitioning and the MSCK REPAIR TABLE command or by enabling partition projection. This makes it the ideal choice for directly querying a large, daily-updated CSV dataset stored in S3.

Exam trap

The trap here is that candidates often confuse AWS Glue's data cataloging and ETL capabilities with interactive querying, or assume Redshift Spectrum is serverless, when in fact it requires a running Redshift cluster.

How to eliminate wrong answers

Option B (Amazon Redshift Spectrum) is wrong because it requires an active Amazon Redshift cluster to be provisioned and running, which adds cost and complexity, and it is designed for querying data in S3 from within Redshift, not as a standalone serverless query service. Option C (Amazon EMR) is wrong because it requires you to provision and manage a cluster of EC2 instances, install Spark or Hive, and submit jobs, which is overkill for simple SQL queries and does not allow direct serverless querying without infrastructure management. Option D (AWS Glue) is wrong because it is primarily a serverless data integration and ETL service, not an interactive SQL query engine; while Glue can catalog data and prepare it for querying, it does not natively execute SQL queries against data in S3 without using Athena or another engine.

Practice this question →
138
MCQeasy

A data engineer needs to analyze large CSV files stored in Amazon S3 using SQL queries. The data is not frequently accessed, and cost is a primary concern. Which AWS service should be used to query the data directly in S3 without moving it?

A.Amazon Athena
B.Amazon EMR
C.Amazon Redshift Spectrum
D.AWS Glue
AnswerA

Athena is serverless and directly queries S3 using SQL with pay-per-query pricing.

Why this answer

Amazon Athena is a serverless interactive query service that allows you to analyze data directly in Amazon S3 using standard SQL, without needing to load or transform the data. It is ideal for this use case because it charges only for the data scanned per query, making it cost-effective for infrequently accessed large datasets. Athena integrates with AWS Glue Data Catalog for schema management and supports common formats like CSV, JSON, Parquet, and ORC.

Exam trap

The trap here is that candidates may choose Amazon Redshift Spectrum because it also queries S3, but they overlook the requirement that it requires a running Redshift cluster, which incurs fixed costs, making it unsuitable for cost-sensitive, infrequent access scenarios.

How to eliminate wrong answers

Option B (Amazon EMR) is wrong because it requires provisioning and managing a cluster of EC2 instances, which incurs ongoing compute costs even when idle, making it less cost-effective for infrequent queries. Option C (Amazon Redshift Spectrum) is wrong because it requires an existing Amazon Redshift cluster to run queries against data in S3, meaning you must pay for the cluster's compute and storage resources regardless of query frequency. Option D (AWS Glue) is wrong because it is primarily an ETL (extract, transform, load) service for preparing and cataloging data, not a direct SQL query engine; while it can catalog data for Athena, using Glue alone to query data would require additional compute resources and is not designed for ad-hoc SQL queries.

Practice this question →
139
Multi-Selecteasy

A data engineering team needs to schedule a nightly ETL job that extracts data from an Amazon RDS for PostgreSQL instance, transforms it using Spark, and loads it into Amazon S3. The team wants to use AWS Glue for this task. Which components are required? (Select TWO.)

Select 2 answers
A.An AWS Glue ETL job with a Spark script.
B.An AWS Glue crawler to populate the Data Catalog.
C.An AWS Glue connection to the RDS database.
D.An AWS Glue development endpoint.
E.An AWS Glue notebook for data exploration.
AnswersA, C

The job performs the defined ETL logic.

Why this answer

An AWS Glue ETL job with a Spark script is required because the transformation step explicitly uses Spark. AWS Glue provides a managed Spark runtime, and the ETL job definition must include a script (either auto-generated or custom) that performs the extract, transform, and load operations. Without this component, the team cannot execute the Spark-based transformation logic.

Exam trap

The trap here is that candidates often assume a crawler is mandatory for any Glue workflow, but the Data Catalog is only needed if you want to use it for schema discovery or as a metastore—the ETL job can operate without it by directly referencing the connection and writing raw data to S3.

Practice this question →
140
MCQhard

A company is running a machine learning training job on Amazon SageMaker that reads training data from an S3 bucket. The job fails intermittently with an S3 throttling error. The data is partitioned across thousands of small files (average 100 KB). Which strategy is MOST effective to resolve the throttling issue?

A.Use Amazon Athena to query the data and output results to a new S3 location
B.Enable S3 Transfer Acceleration on the bucket
C.Combine the small files into larger files (e.g., 100 MB) using a preprocessing step
D.Increase the number of SageMaker training instances to distribute the load
AnswerC

Larger files reduce the number of GET requests, mitigating throttling.

Why this answer

S3 throttling errors (HTTP 503) occur when many small files cause a high request rate per prefix. By combining thousands of 100 KB files into fewer 100 MB files, you drastically reduce the number of GET requests, staying within S3's 5,500 GET requests per second per prefix limit. This preprocessing step directly addresses the root cause of the throttling without changing the training infrastructure.

Exam trap

The trap here is that candidates confuse network-level optimizations (Transfer Acceleration) or parallelization (more instances) with the fundamental S3 request rate limit, which is a per-prefix throughput constraint, not a bandwidth issue.

How to eliminate wrong answers

Option A is wrong because Athena also issues GET requests to S3 and would itself be throttled or create additional overhead without consolidating the files; it does not reduce the number of small objects. Option B is wrong because S3 Transfer Acceleration optimizes network latency for uploads/downloads over long distances, not the request rate per prefix; it does not mitigate throttling caused by too many small files. Option D is wrong because increasing training instances multiplies the number of concurrent GET requests, worsening the throttling issue rather than resolving it.

Practice this question →
141
Matchingmedium

Match each data format to its typical use in AWS ML.

Drag a concept onto its matching description — or click a concept then click the description.

Concepts
Matches

Tabular data for SageMaker built-in algorithms

Efficient binary format for SageMaker

Columnar storage for analytics

Semi-structured data, e.g., for Lambda

TensorFlow training data format

Why these pairings

Common data formats in AWS ML: CSV for tabular data, JSON for semi-structured inference I/O, Parquet for columnar analytics, and TFRecord for TensorFlow training. Distractors confuse CSV with Parquet and JSON with binary formats.

Practice this question →
142
MCQhard

A data engineer is troubleshooting an AWS Glue job that reads from and writes to the S3 bucket 'data-lake-bucket'. The job fails when trying to write to the 'sensitive/' prefix. The IAM policy attached to the Glue job's IAM role is shown in the exhibit. What is the MOST likely reason for the failure?

A.The IAM role does not have permission to read objects from the bucket
B.The IAM role has an explicit deny for s3:PutObject on the 'sensitive/' prefix
C.The IAM policy does not specify the bucket resource correctly
D.The IAM policy lacks a required condition for encryption
AnswerB

The Deny statement blocks write access to the sensitive prefix.

Why this answer

Even though the first statement allows s3:PutObject on the entire bucket, the second statement explicitly denies s3:PutObject on the 'sensitive/' prefix. Explicit deny overrides any allow. Option A is wrong because the policy allows GetObject.

Option C is wrong because the policy covers the bucket. Option D is wrong because there is a deny statement.

Practice this question →
143
MCQhard

A data engineer needs to move 10 TB of historical data from an on-premises Hadoop cluster to Amazon S3 for ML training. The data is currently stored in HDFS and is compressible. The network bandwidth between the on-premises data center and AWS is 1 Gbps. The team needs to minimize the time to transfer and also wants to avoid any downtime for the on-premises system. Which solution meets these requirements?

A.Set up an AWS Direct Connect connection and use rsync to copy data to S3.
B.Enable S3 Transfer Acceleration on the bucket and use the AWS CLI to copy data.
C.Install the AWS DataSync agent on-premises, configure a task to transfer data to S3 with compression enabled.
D.Use AWS Snowball Edge devices to export the data and ship them to AWS.
AnswerC

DataSync is optimized for large data transfers with compression and parallelization.

Why this answer

AWS DataSync is designed for large-scale data transfers from on-premises storage to AWS, and it can compress data in transit to reduce transfer time over a 1 Gbps link. It also operates as an agent-based solution that does not require downtime for the on-premises Hadoop cluster, as it reads data from HDFS without disrupting ongoing operations.

Exam trap

The trap here is that candidates often overlook the compression capability of DataSync and assume that faster network options like Direct Connect or Transfer Acceleration alone are sufficient, ignoring that compression is critical to minimize transfer time over a fixed bandwidth.

How to eliminate wrong answers

Option A is wrong because rsync over Direct Connect does not natively compress data during transfer, and without compression, transferring 10 TB over 1 Gbps would take over 22 hours, failing to minimize time. Option B is wrong because S3 Transfer Acceleration optimizes network path but does not compress data; the raw 10 TB transfer over 1 Gbps still takes too long, and it requires the on-premises system to be actively serving data, which could cause downtime if not carefully managed. Option D is wrong because Snowball Edge involves physical shipping, which introduces days of latency and is not the fastest option for 10 TB when a 1 Gbps network is available; it also requires exporting data from HDFS to the device, which can cause downtime if not properly orchestrated.

Practice this question →
144
MCQhard

A data engineering team is building a real-time data pipeline using Amazon Kinesis Data Streams with AWS Lambda for processing. The pipeline ingests clickstream data from a mobile app. The team notices that occasionally, a Lambda function fails due to a transient error, and the failed record is not retried, leading to data loss. The Lambda function is configured with a batch size of 100 and a maximum retry count of 0. The team wants to ensure that all records are processed successfully, even if transient failures occur. They also want to minimize the impact of poison pill records that could block processing. Which combination of actions should the team take to address this issue?

A.Set the maximum retry count to 5 and configure a dead-letter queue on the Lambda function to capture failed records after retries.
B.Switch to using Amazon Kinesis Data Firehose to buffer data and use AWS Lambda for transformation with built-in retry logic.
C.Set the maximum retry count to 5, configure an on-failure destination Amazon SQS queue, and set up a dead-letter queue on that SQS queue for poison pills.
D.Reduce the batch size to 1 and increase the Lambda function timeout to handle transient errors.
AnswerC

This provides retries and isolates poison pills without blocking the main stream.

Why this answer

To address the issue of data loss due to transient errors and poison pill records, the team should increase the Lambda function's maximum retry count to 5 to allow retries on transient failures. However, even with retries, some records may fail repeatedly (poison pills) which can block the shard if not handled. Configuring an on-failure destination (such as an Amazon SQS queue) on the Lambda function sends all records that failed after retries to that queue.

Then, by setting up a dead-letter queue on that SQS queue, poison pill records are isolated and can be examined or reprocessed separately, preventing them from blocking the main processing pipeline. Option A is incorrect because a dead-letter queue on Lambda alone is not sufficient – it captures failures after retries if configured, but the key is to also have an on-failure destination to offload failures. Option B is incorrect because Kinesis Data Firehose is designed for streaming data to destinations like S3, not for real-time per-record Lambda processing with built-in retry logic; it would change the architecture.

Option D is incorrect because reducing batch size to 1 would increase costs and processing time, and may not fully resolve transient errors or poison pill issues.

Practice this question →
145
MCQhard

A financial services company is building a fraud detection model that requires joining real-time transaction data with a reference dataset of known fraudulent accounts stored in Amazon DynamoDB. The solution must minimize latency and be highly available. The reference dataset is updated frequently (every few minutes). Which architecture should the team use?

A.Use Amazon Athena to query the DynamoDB table and join with streaming data.
B.Use Amazon Kinesis Data Analytics to process the stream and join with a DynamoDB table.
C.Use AWS Glue streaming ETL to read from Kinesis and join with DynamoDB.
D.Use Amazon SageMaker to host a model that queries DynamoDB for each inference.
AnswerB

Kinesis Data Analytics supports real-time joins with DynamoDB using reference data.

Why this answer

Amazon Kinesis Data Analytics (now managed Apache Flink) can directly reference a DynamoDB table as a reference source via the Flink Table API or SQL JOINs, enabling low-latency, stateful stream enrichment without external query overhead. This architecture minimizes latency by performing the join in-memory within the streaming application, and it supports high availability through Kinesis Data Analytics' automatic checkpointing and failover.

Exam trap

The trap here is that candidates often choose AWS Glue streaming ETL (Option C) because they associate Glue with ETL and DynamoDB, but Glue streaming ETL lacks native DynamoDB reference join support, making Kinesis Data Analytics the correct low-latency streaming join service.

How to eliminate wrong answers

Option A is wrong because Amazon Athena is an interactive query service designed for ad-hoc analytics on data in S3, not for real-time stream processing; querying DynamoDB via Athena would introduce high latency and cannot continuously join with streaming data. Option C is wrong because AWS Glue streaming ETL reads from Kinesis but does not natively support joining with a DynamoDB table as a reference source; it would require custom workarounds like reading DynamoDB into a Spark DataFrame, adding latency and complexity. Option D is wrong because hosting a model on SageMaker and querying DynamoDB for each inference introduces network round-trip latency per request, which is unacceptable for real-time fraud detection at high throughput, and SageMaker endpoints are not designed for frequent external database lookups.

Practice this question →
146
MCQhard

A data engineer runs the CLI command to download an object from S3. The bucket owner is 123456789012, and the engineer's IAM user has s3:GetObject permission on the bucket. The object was uploaded by a different AWS account. What is the MOST likely reason for the AccessDenied error?

A.The --expected-bucket-owner parameter is incorrect
B.The object is owned by a different AWS account, and the bucket owner has not been granted access
C.The bucket policy denies access to the engineer's IAM user
D.The IAM policy does not allow s3:GetObject for that specific key
AnswerB

Object ACLs or bucket policy must grant access to bucket owner.

Why this answer

When an object is uploaded to S3 by a different AWS account, the object is owned by the uploading account, not the bucket owner. By default, the bucket owner does not have access to objects uploaded by other accounts, even if the bucket owner has a policy granting s3:GetObject to their IAM users. The engineer's IAM user has permission on the bucket, but the object itself is not owned by the bucket owner, so the bucket owner cannot delegate access to it unless the object owner explicitly grants read access via an object ACL or a bucket policy that the object owner accepts.

Exam trap

The trap here is that candidates assume bucket-level permissions (like s3:GetObject on the bucket) automatically grant access to all objects in the bucket, but S3's object ownership model requires explicit permission from the object owner for objects uploaded by other accounts.

How to eliminate wrong answers

Option A is wrong because the --expected-bucket-owner parameter is used to ensure the bucket owner matches the expected account ID, but it does not cause an AccessDenied error; it would cause a different error (e.g., 'BucketOwnerMismatch') if the bucket owner does not match. Option C is wrong because the bucket policy does not deny access; the error arises from the object ownership model, not from an explicit deny in the bucket policy. Option D is wrong because the IAM policy does allow s3:GetObject for the bucket, but the issue is that the object is owned by a different account, and the bucket owner (and thus the engineer) lacks access rights to that specific object.

Practice this question →
147
MCQhard

A data engineer is designing a data lake on Amazon S3 that must support both batch and streaming analytics. The data comes in Parquet format and needs to be queryable by Amazon Athena. Which partitioning strategy will optimize query performance and reduce costs?

A.Partition by date and hour for time-based queries
B.Store data as CSV without partitioning for simplicity
C.Partition by device_id for granular access
D.Use a single partition for all data to simplify management
AnswerA

Common query patterns are time-filtered; this reduces data scanned.

Why this answer

Partitioning by date and hour is optimal for time-series data in Parquet format queried by Athena because it leverages Hive-style partitioning to prune partitions during query execution, drastically reducing the amount of data scanned. This minimizes Athena's cost (which is based on data scanned) and improves query performance by limiting I/O to only the relevant partitions. Parquet's columnar storage further reduces scan volume when queries select only specific columns, making this combination highly efficient for both batch and streaming ingestion patterns.

Exam trap

AWS often tests the misconception that high-cardinality partitions (like device_id) improve query performance, but in Athena and Presto, they actually degrade performance due to excessive partition metadata and small file overhead, whereas coarse-grained time partitions are the recommended pattern.

How to eliminate wrong answers

Option B is wrong because storing data as CSV without partitioning forces full-table scans for every query, increasing Athena's cost (data scanned) and degrading performance, while CSV lacks the compression and columnar benefits of Parquet. Option C is wrong because partitioning by device_id creates an excessive number of small partitions (high cardinality), leading to metadata overhead, slow partition discovery, and poor query performance in Athena, which is optimized for coarse-grained, time-based partitioning. Option D is wrong because using a single partition for all data eliminates the benefits of partition pruning, causing Athena to scan the entire dataset for every query, which is both expensive and slow.

Practice this question →
148
MCQeasy

A company uses Amazon Kinesis Data Streams to collect clickstream data. The data is consumed by a Lambda function that writes to DynamoDB. Occasionally, the Lambda function fails due to throttling from DynamoDB. How can the company resolve this issue without losing data?

A.Ignore the throttling errors and let Lambda retry.
B.Increase the number of shards in the Kinesis stream.
C.Use an Amazon SQS queue as a buffer between Kinesis and Lambda.
D.Decrease the batch size in the Lambda event source mapping.
AnswerD

Smaller batches reduce the write rate, avoiding throttling.

Why this answer

Decreasing the batch size in the Lambda event source mapping reduces the number of records sent to each Lambda invocation. This lowers the write throughput demand on DynamoDB per invocation, mitigating throttling while still allowing Lambda to retry failed records individually. The Kinesis stream retains data for up to 365 days, so no data is lost as long as the Lambda function eventually processes all records.

Exam trap

The trap here is that candidates often assume increasing shards or adding a buffer will solve throttling, but the real issue is the downstream write volume per invocation, which is directly controlled by the batch size in the event source mapping.

How to eliminate wrong answers

Option A is wrong because ignoring throttling errors and relying solely on Lambda retries can lead to repeated failures, increased latency, and potential data loss if the retry policy is exhausted or the event source mapping discards records after a maximum retry count. Option B is wrong because increasing the number of shards in the Kinesis stream increases the parallelism and throughput of data ingestion, but it does not address the downstream DynamoDB throttling; it may actually worsen the problem by sending more data to Lambda faster. Option C is wrong because using an SQS queue as a buffer between Kinesis and Lambda adds unnecessary complexity and latency, and Kinesis Data Streams already provides durable storage with per-record retry logic; SQS does not solve the root cause of DynamoDB throttling.

Practice this question →
149
MCQmedium

A research institution is building a data lake to store genomics data. Each experiment generates multiple files totaling about 500 GB. The data is stored in Amazon S3 and needs to be processed by multiple machine learning (ML) training jobs running on Amazon SageMaker. The data has a high churn rate; after 30 days, most data becomes irrelevant and should be moved to Amazon S3 Glacier Deep Archive. The institution wants to minimize storage costs while maintaining data durability. Which S3 storage class should they use for the first 30 days?

A.Use S3 Intelligent-Tiering for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.
B.Use S3 One Zone-IA for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.
C.Use S3 Standard for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.
D.Use S3 Glacier Instant Retrieval for all data, and set a lifecycle policy to transition to S3 Glacier Deep Archive after 30 days.
AnswerA

Intelligent-Tiering automatically optimizes costs by moving data to lower-cost tiers when not accessed, and it provides high durability.

Why this answer

S3 Intelligent-Tiering is the most cost-effective storage class for the first 30 days because it automatically moves data between frequent and infrequent access tiers based on usage, without any retrieval fees. This is ideal for genomics data that may have unknown or changing access patterns during the initial processing period. After 30 days, a lifecycle rule transitions the data to S3 Glacier Deep Archive for long-term storage, minimizing costs.

S3 Standard is more expensive for data that may not be accessed frequently, S3 One Zone-IA lacks durability across Availability Zones, and S3 Glacier Instant Retrieval is designed for long-lived, rarely accessed data and is not cost-effective for the first 30 days.

Practice this question →
150
MCQeasy

A data engineer is designing a data lake on Amazon S3. The data comes from various sources, including IoT devices, web logs, and transactional databases. The engineer needs to organize the data in a way that supports efficient querying using Amazon Athena and allows for easy management of access permissions. Which S3 bucket structure is the most appropriate?

A.Store all data in a single prefix without any partitioning.
B.Use a prefix structure like s3://bucket/source/year/month/day/.
C.Store all data in separate S3 buckets for each source and date.
D.Use a prefix structure like s3://bucket/date/source/.
AnswerB

This structure enables partition pruning by source and time, optimizing Athena queries and allowing granular access control at the source level.

Why this answer

Partitioning by source, year, month, day allows Athena to prune partitions, reducing scan costs and improving performance. Option A is wrong because storing all data in a flat structure forces full scans. Option C is wrong because prefix-based access controls can be applied at the source level within the partitioned structure.

Option D is wrong because using date as the first partition level is less intuitive for managing permissions by source.

Practice this question →
← PreviousPage 2 of 5 · 350 questions totalNext →

Ready to test yourself?

Try a timed practice session using only Data Engineering questions.

Start 20-question session