DEA-C01 Mock Exam 97 — 25 Questions

Question 1

Your company has an Amazon S3-based data lake partitioned by year/month/day. An AWS Glue crawler runs daily to update the Data Catalog. A Spark job on Amazon EMR reads the latest partition and performs transformations. Recently, the Spark job has been failing with a 'FileNotFoundException' for a file that is expected to exist. You check the S3 bucket and see that the file exists. The job is configured to use S3 as the direct input source with EMRFS consistent view enabled. The IAM role for the EMR cluster has full S3 access. What is the most likely cause?

Accepted Answer

The EMR cluster is experiencing eventual consistency issues despite EMRFS consistent view.. Option C is correct because even with EMRFS consistent view enabled, there can be a propagation delay between the crawler updating the catalog and the actual S3 objects being visible. The Spark job may be reading stale metadata. Option A is wrong because the crawler updates the catalog, but the job may read from S3 directly. Option B is wrong because the file exists. Option D is wrong because the catalog is updated.

Answer

The Data Catalog partition metadata is out of sync because the crawler runs after the job.

Answer

The Glue crawler did not update the Data Catalog because of insufficient permissions.

Answer

The file was deleted by a lifecycle policy after the job started.

Question 2

A data engineer at a financial services company manages an AWS Glue ETL pipeline that processes transaction data from Amazon S3 to Amazon Redshift for reporting. The pipeline runs every hour and uses a Glue job that reads Parquet files, performs transformations in Spark, and writes to Redshift using the JDBC connector. Recently, the job has been failing intermittently with the error: 'java.sql.BatchUpdateException: ERROR: null value in column "transaction_id" violates not-null constraint'. The data engineer has verified that the source Parquet files do contain non-null values for transaction_id. The job uses a DynamicFrame and applies a mapping to rename columns. The engineer also noticed that the failure occurs only during peak hours when there is high concurrency on Redshift. Which course of action should the engineer take to resolve this issue?

Accepted Answer

Review the Glue job's mapping transformation to ensure transaction_id is correctly mapped and not dropped.. Option C is correct. The error suggests that some rows are being written with null transaction_id. During high concurrency, Redshift might be rejecting the batch due to a transient issue, but the error is about null constraint. The most likely cause is that the mapping is incorrectly dropping or nullifying the column. Option A is wrong because increasing Glue's worker count does not address the null value issue. Option B is wrong because increasing Redshift WLM concurrency could exacerbate the problem. Option D is wrong because the source files are not the issue.

Answer

Add a filter in Glue to remove rows with null transaction_id.

Answer

Increase the Redshift WLM concurrency scaling to handle more queries.

Answer

Increase the number of Glue workers to handle peak-hour load.

Question 3

A data engineer is troubleshooting an AWS Glue ETL job that uses a Python shell script to extract data from an Amazon RDS for PostgreSQL database and load it into an Amazon Redshift table. The job runs successfully, but the data engineer notices that the row count in Redshift is consistently lower than the row count in PostgreSQL. The job uses a SELECT * query without any filtering. The data engineer suspects that some rows are being dropped during the transfer. The job uses the psycopg2 library to connect to PostgreSQL and the psycopg2 connection is configured with autocommit=True. The Redshift table has no constraints that would reject rows. What is the most likely cause of the missing rows?

Accepted Answer

The SELECT * query includes columns with data types that are not supported by psycopg2.. Option C is correct. If the Redshift table is set to distribute data, and if the distribution key is not unique, some rows may be lost if there are duplicates? Actually, the most common cause is that the Glue job is using multiple executors, and the data is being split across them, but Python shell uses a single executor. However, the most plausible answer is that the SELECT * query may include columns with special characters that are not handled correctly. Option A is wrong because autocommit=True should not cause data loss. Option B is wrong because SSL/TLS is about encryption, not row count. Option D is wrong because the Redshift table has no constraints.

Answer

The SSL/TLS connection to PostgreSQL is dropping packets.

Answer

The autocommit=True setting is causing incomplete transactions.

Answer

The Redshift table has a distribution key that causes some rows to be silently discarded.

Question 4

A company uses Amazon S3 to store log files from multiple applications. The logs are written in JSON format. A data engineer wants to use Amazon Athena to query these logs. The logs are stored in a bucket with the following structure: 's3://logs/app1/date=2021-01-01/'. The engineer creates an Athena table with partitions. However, when querying, Athena returns zero results for partitions that exist. The engineer has run MSCK REPAIR TABLE to add partitions. What is the most likely cause of the issue?

Accepted Answer

The partition key name in the table definition does not match the S3 folder naming convention.. Option C is correct because Athena relies on the partition metadata stored in the Glue Data Catalog. If the partition folder structure does not match the table's partition definition (e.g., the folder is named 'date=2021-01-01' but the table's partition key is named 'dt'), MSCK REPAIR will not register the partitions. The partition key name must match the folder prefix. Option A is wrong because Athena supports JSON format. Option B is wrong because MSCK REPAIR does add partitions if the structure matches. Option D is wrong because the data is already in the S3 bucket; no need to copy.

Answer

The MSCK REPAIR TABLE command failed silently.

Answer

The log files are in JSON format and Athena does not support JSON.

Answer

The log files need to be copied to a different bucket in the same region.

Question 5

Your organization uses Amazon Redshift for analytical workloads. You have noticed that queries are slow on a large fact table. The table is distributed by KEY on the customer_id column and sorted by transaction_date. The table is frequently updated with new records. To improve query performance, you decide to implement a distribution style that reduces data movement. Which action should you take?

Accepted Answer

Change the distribution style to EVEN to distribute rows evenly across all nodes.. Option D is correct because EVEN distribution distributes rows evenly across nodes, reducing data movement for queries that do not benefit from key distribution. Option A is wrong because ALL distribution duplicates the entire table on each node, which is inefficient for large tables. Option B is wrong because AUTO lets Redshift decide, but it may not choose the best. Option C is wrong because sorting by other columns does not reduce data movement.

Answer

Change the distribution style to ALL to put a copy of the table on every node.

Answer

Change the distribution style to AUTO to let Redshift choose the best distribution.

Answer

Change the sort key to include customer_id as well.

Question 6

A company uses Amazon Kinesis Data Firehose to deliver streaming data to an Amazon S3 bucket. The data is then processed by a scheduled AWS Glue ETL job that loads it into an Amazon Redshift table. Recently, the Glue job has been failing with the error: 'S3ServiceException: Access Denied'. The Firehose delivery stream is configured with a prefix and error logging to the same S3 bucket. The Glue job uses the same IAM role that has s3:GetObject and s3:ListBucket permissions on the bucket. What is the most likely cause?

Accepted Answer

The S3 bucket has default encryption enabled with AWS KMS (SSE-KMS), and the Glue job's IAM role lacks kms:Decrypt permission.. Option D is correct because Firehose uses SSE-S3 by default unless configured otherwise. If the S3 bucket has default encryption enabled with SSE-KMS, Firehose will use that encryption, but the Glue job's IAM role may lack kms:Decrypt permission for the KMS key. The error 'Access Denied' when reading from S3 often indicates encryption permission issues. Option A is wrong because the Glue job can read from S3 with the current permissions if no encryption is involved. Option B is wrong because the error is about access, not schema. Option C is wrong because the Glue job can use the same role as Firehose, but the role may not have KMS permissions.

Answer

The Glue job expects a different data format than what Firehose writes.

Answer

The Glue job's IAM role does not have s3:GetObjectVersion permission.

Answer

The Glue job is using the wrong IAM role that does not have permissions to the S3 bucket.

Question 7

A data engineer is responsible for a data pipeline that uses Amazon S3 as a data lake, AWS Glue for ETL, and Amazon Athena for ad-hoc queries. The pipeline ingests CSV files from an external partner via SFTP into an S3 bucket. The files are then processed by a Glue job that converts them to Parquet and writes to a separate S3 bucket partitioned by date. The Glue job runs daily and is triggered by a scheduled CloudWatch Events rule. Recently, the data engineer noticed that some days the Glue job fails because of memory errors, and on those days the Athena queries that rely on the data return incomplete results. The engineer needs to ensure that the pipeline is resilient and that Athena queries always see a complete view of the data, even if the Glue job fails mid-run. The engineer also needs to minimize re-processing of data. Which course of action should the engineer take?

Accepted Answer

Modify the Glue job to use job bookmarks for incremental processing and write the Parquet output to a temporary location, then use an S3 copy operation to move the data into the final partitioned location only after the job completes successfully.. Option B is correct. Using Glue job bookmarks enables incremental processing and the ability to resume from the last successful checkpoint. Staging the data in a temporary location and moving it atomically ensures that Athena sees only complete data. Option A is wrong because increasing worker capacity does not prevent partial writes. Option C is wrong because using Lambda for conversion is less scalable and error-prone. Option D is wrong because partition projection does not solve the atomicity issue.

Answer

Increase the number of workers and the worker type to G.2X to handle the memory errors, and enable job retries.

Answer

Replace the Glue job with an AWS Lambda function that processes the CSV files and writes Parquet to S3, and use S3 Event Notifications to trigger the function.

Answer

Use Athena partition projection to automatically discover partitions and set up a retry mechanism using AWS Step Functions.

Question 8

A data engineer is responsible for a real-time data pipeline that ingests clickstream data from a website into Amazon Kinesis Data Streams, then processed by an AWS Lambda function that writes to an Amazon DynamoDB table for user session tracking. The Lambda function is idempotent and uses the DynamoDB PutItem API with a condition expression to avoid overwriting existing records. Over the past week, the engineer has observed an increase in DynamoDB write throttling (ProvisionedThroughputExceededException) during peak traffic hours. The DynamoDB table has on-demand capacity. The engineer checks the Lambda function's reserved concurrency and finds it set to 1000. The Kinesis stream has 10 shards. The Lambda function's batch size is set to 100. The engineer suspects that the retry behavior is causing duplicate writes and throttling. Which change should the engineer make to reduce throttling?

Accepted Answer

Decrease the Lambda batch size to 10 to reduce the number of records processed per invocation.. Option B is correct. On-demand DynamoDB can scale, but it has a per-partition throughput limit. Reducing the Lambda batch size reduces the number of concurrent writes per shard, decreasing the chance of hitting partition limits. Option A is wrong because increasing shards would increase concurrency, worsening throttling. Option C is wrong because decreasing reserved concurrency could cause Lambda throttling but not DynamoDB throttling. Option D is wrong because using a DynamoDB stream adds complexity and does not directly reduce write throttling.

Answer

Increase the number of Kinesis shards to 20 to distribute the load.

Answer

Decrease the Lambda reserved concurrency to 500 to limit the number of concurrent invocations.

Answer

Use a DynamoDB Stream to trigger a second Lambda function that writes to the table.

Question 9

A company uses Amazon S3 to store raw data and AWS Glue to run ETL jobs. The data is partitioned by date in the format 'year=YYYY/month=MM/day=DD'. A new data source started sending data with a different date format 'YYYY-MM-DD'. The Glue crawler is configured to create a single table for the entire bucket. The crawler runs daily, but it is not detecting the new partitions from the new data source. The existing partitions are in the format 'year=2024/month=05/day=10', while the new data is stored as '2024-05-10/' without the key-value structure. How should the engineer modify the data pipeline to include the new data?

Accepted Answer

Modify the new data source to store data in the same Hive-style partition format as the existing data.. Option A is correct because the new data uses a different partition structure. The crawler expects Hive-style partitions (key=value). To include the new data, the engineer should either change the folder structure to match the existing one or configure a separate crawler for the new format. Option B is wrong because the crawler will not automatically infer non-Hive partitions. Option C is wrong because the data format (CSV, JSON) is not the issue. Option D is wrong because Glue does not have a 'partition discovery' setting that overrides the structure.

Answer

Run the crawler with the 'Create partition indexes' option enabled.

Answer

Configure the crawler to add a custom classifier for date formats.

Answer

Convert the new data to Parquet format.

Question 10

A company uses Amazon DynamoDB as a data store for a real-time dashboard application. The application performs point lookups and range queries on a table that has a partition key and sort key. The table uses on-demand capacity mode. Recently, the application's response time has increased, and CloudWatch metrics show high 'ThrottledRequests' for the table. The application uses the AWS SDK with default retry settings. The data access pattern is read-heavy with occasional spikes. What is the most effective way to reduce throttling?

Accepted Answer

Implement exponential backoff with jitter in the application code.. Option C is correct because DynamoDB on-demand mode accommodates traffic spikes but can throttle if the spike exceeds the previous peak by more than double. Implementing exponential backoff with jitter in the application allows retries to succeed without overwhelming the table. Option A is wrong because switching to provisioned capacity would require predicting the peak and may still throttle if the spike is higher. Option B is wrong because DAX (DynamoDB Accelerator) is an in-memory cache that reduces read load but does not eliminate throttling from write spikes; it also adds cost and complexity. Option D is wrong because increasing read capacity units only applies to provisioned mode.

Answer

Switch the table to provisioned capacity and set the read capacity units to a high value.

Answer

Enable DynamoDB Accelerator (DAX) to cache frequently read items.

Answer

Increase the read capacity units to a higher value.

Question 11

A company runs a data lake on Amazon S3 with AWS Lake Formation for access control. The data lake contains sensitive customer information. A data scientist needs to query the data using Amazon Athena. The data scientist has been granted SELECT permission on the database and tables via Lake Formation. However, when the data scientist runs a query in Athena, they receive an error: 'Access denied. Please check your permissions.' The IAM role used by Athena has the following permissions: s3:GetObject, s3:ListBucket, and lakeformation:GetDataAccess. The Lake Formation admin has verified that the data scientist is a member of a Lake Formation data lake location and has been granted 'Describe' and 'Select' permissions on the table. What is the most likely reason for the access denied error?

Accepted Answer

The S3 bucket policy does not grant the Athena IAM role access to the S3 location.. Option A is correct because Lake Formation requires the IAM role to have lakeformation:GetDataAccess permission, but also the role needs permissions to the underlying S3 location. If the S3 bucket policy does not allow the Athena IAM role to access the bucket, the request will be denied. The data scientist is granted permissions in Lake Formation, but the S3 bucket policy must also grant access to the IAM role. Option B is wrong because the role already has lakeformation:GetDataAccess. Option C is wrong because the error is not about Lake Formation tag access. Option D is wrong because Athena uses the IAM role's permissions, not the user's IAM permissions directly.

Answer

The data scientist is not assigned to the correct Lake Formation tag.

Answer

The Athena IAM role is missing lakeformation:GetEffectivePermissions permission.

Answer

The data scientist's IAM user lacks the necessary S3 permissions.

Question 12

Your team uses Amazon Kinesis Data Analytics to process real-time streaming data from an Amazon Kinesis Data Stream. The application calculates windowed aggregations and writes results to an Amazon S3 bucket using a delivery stream. Recently, the application has been failing with a 'LimitExceededException' when writing to the delivery stream. You have checked the CloudWatch metrics and see that the IncomingBytes and IncomingRecords for the delivery stream are well below the provisioned limits. The delivery stream has a buffer size of 5 MB and a buffer interval of 60 seconds. The application generates about 500 records per second, each about 1 KB. What is the most likely cause and correct action?

Accepted Answer

Modify the application to use PutRecordBatch with smaller batch sizes to stay within the 4 MB per-call limit.. Option B is correct because Kinesis Data Analytics writes to Firehose through a PutRecord or PutRecordBatch call. Each call has a maximum payload size of 1 MB (for PutRecord) or 4 MB (for PutRecordBatch). If the application uses PutRecordBatch and the total payload exceeds 4 MB, it gets a LimitExceededException. Increasing the buffer size or interval does not affect the per-call limit. Option A is wrong because the stream is not the source of the error. Option C is wrong because the buffer settings are not causing the per-call limit. Option D is wrong because the data size is small.

Answer

Increase the number of shards in the Kinesis Data Stream to reduce the load on the application.

Answer

Reduce the buffer interval of the delivery stream to 30 seconds to flush data more frequently.

Answer

Increase the buffer size of the delivery stream to 10 MB to accommodate larger writes.

Question 13

A company uses AWS Glue DataBrew to clean and prepare data for machine learning. The source data is in an S3 bucket with server-side encryption using AWS KMS (SSE-KMS). The DataBrew project is set up with an IAM role that has permissions to read from the S3 bucket and use the KMS key. When the DataBrew job runs, it fails with an error indicating that it cannot access the data. The IAM role has the following policy: { 'Version': '2012-10-17', 'Statement': [ { 'Effect': 'Allow', 'Action': ['s3:GetObject', 's3:ListBucket'], 'Resource': ['arn:aws:s3:::my-bucket', 'arn:aws:s3:::my-bucket/*'] }, { 'Effect': 'Allow', 'Action': 'kms:Decrypt', 'Resource': 'arn:aws:kms:us-east-1:123456789012:key/my-key' } ] }. What is the most likely cause of the failure?

Accepted Answer

The IAM role is missing s3:PutObject permission on the DataBrew output bucket.. Option C is correct because DataBrew uses a separate S3 bucket for storing intermediate outputs and recipe results. The IAM role needs s3:PutObject permission on that bucket. The error typically manifests as access denied when DataBrew tries to write. Option A is wrong because the role has kms:Decrypt permission. Option B is wrong because DataBrew does not require VPC endpoints by default. Option D is wrong because the role includes s3:ListBucket.

Answer

The IAM role is missing s3:ListBucket permission on the source bucket.

Answer

The S3 bucket is in a different region than the DataBrew project.

Answer

The IAM role is missing kms:GenerateDataKey permission for the KMS key.

Question 14

A company runs a data pipeline that uses Amazon EMR to process large datasets. The pipeline reads data from S3, processes it using Spark, and writes results back to S3. Recently, the pipeline has been failing with 'OutOfMemoryError' in the Spark executors. The EMR cluster is configured with 5 core nodes of type m5.xlarge (4 vCPU, 16 GB memory each). The Spark application uses dynamic allocation and default Spark configurations. The input data size is approximately 500 GB in Parquet format. What is the most cost-effective way to resolve the out-of-memory errors?

Accepted Answer

Change the core node instance type to r5.xlarge (32 GB memory) and keep 5 nodes.. Option B is correct because the current cluster has limited memory per node (16 GB). By switching to memory-optimized instances like r5.xlarge (32 GB), each node has double the memory, reducing the chance of OOM. This is more cost-effective than adding more nodes because the total memory per node increases without increasing the number of instances. Option A is wrong because increasing the number of nodes adds more memory but also more cost; it might be more expensive than using fewer, larger nodes. Option C is wrong because it's generally not recommended to increase spark.executor.memory beyond the physical memory; it could cause YARN to kill containers. Option D is wrong because Parquet is already efficient; changing to a different format may not solve memory issues.

Answer

Increase the spark.executor.memory setting to 8 GB in the Spark configuration.

Answer

Increase the number of core nodes to 10 to distribute the data across more executors.

Answer

Change the input data format from Parquet to ORC to reduce memory footprint.

Question 15

A company runs a data pipeline that ingests streaming data from an IoT fleet into Amazon Kinesis Data Streams (KDS) with 50 shards. A Lambda function processes records from the stream and writes them to an Amazon DynamoDB table for real-time analytics. The Lambda function is configured with a batch size of 100 and a maximum batching window of 60 seconds. Recently, the company has been seeing an increasing number of 'WriteProvisionedThroughputExceededException' errors from DynamoDB, causing Lambda to retry and eventually send records to a dead-letter queue (DLQ). The DynamoDB table is provisioned with 5000 read capacity units (RCU) and 5000 write capacity units (WCU). The average item size is 1 KB. The KDS stream receives an average of 8000 records per second, each 2 KB in size. The Lambda function performs a simple transformation and writes each record individually to DynamoDB. The company wants to reduce the throttling errors without increasing the DynamoDB WCU provision. Which course of action is most likely to achieve this?

Accepted Answer

Modify the Lambda function to use DynamoDB BatchWriteItem to write records in batches of 25.. Option A is correct because writing records in batches using DynamoDB's BatchWriteItem API reduces the number of write requests, lowering the consumed WCU per request (since each batch consumes WCU for all items but with fewer API calls, reducing overhead). This can reduce throttling without increasing WCU. Option B is wrong because increasing Lambda batch size would cause more records to be processed per invocation, but if each record is still written individually, the number of write requests remains the same. Option C is wrong because increasing Lambda concurrency would increase the number of concurrent invocations, potentially increasing throttling. Option D is wrong because increasing Lambda timeout does not affect the rate of writes.

Answer

Increase the Lambda function's reserved concurrency to 1000.

Answer

Increase the Lambda function timeout to 5 minutes to allow more time for retries.

Answer

Increase the Lambda batch size to 500 and reduce the batching window to 30 seconds.

Question 16

A company uses Amazon Kinesis Data Streams to ingest clickstream data from web applications. The data is consumed by a Kinesis Data Analytics application that runs SQL queries. Recently, the data analytics application has been falling behind, and the 'MillisBehindLatest' metric for the consumer has been increasing steadily. The shard count is 4, and the average records per second per shard is 200, with an average record size of 1 KB. The provisioned shard limit for the account is 10. Which action will resolve the issue?

Accepted Answer

Increase the number of shards in the Kinesis stream to 8.. Option C is correct because the consumer is falling behind due to insufficient read capacity. Increasing the number of shards increases the total read capacity and allows the consumer to keep up. Option A is wrong because the write capacity is not the issue; the consumer is behind. Option B is wrong because switching to enhanced fan-out does not address the shard count limitation; it improves dedicated throughput per consumer but the total throughput is still limited by shard count. Option D is wrong because the consumer is already using KCL, and the issue is not related to checkpointing.

Answer

Enable enhanced fan-out on the Kinesis stream and subscribe the analytics application to it.

Answer

Reduce the checkpoint interval on the Kinesis Client Library (KCL) consumer to commit offsets more frequently.

Answer

Increase the provisioned write capacity of the Kinesis stream by requesting a shard limit increase.

Question 17

A data engineer is troubleshooting an Amazon Redshift cluster that is running slowly. The cluster has 4 dc2.large nodes. The engineer runs a query that scans a large table and notices that the query uses only a single slice instead of all slices. The table is distributed with DISTSTYLE ALL. What is the most likely reason for the query using only one slice?

Accepted Answer

The table uses DISTSTYLE ALL, which stores the entire table on a single slice per node.. Option B is correct because DISTSTYLE ALL means the entire table is copied to every node, but within each node, the data is stored on a single slice. Redshift slices are determined by the number of cores per node; for dc2.large nodes, there are 2 slices per node. However, with DISTSTYLE ALL, each node stores a full copy of the table, but the table is not distributed across slices; it is replicated to each node, and queries may not parallelize across slices efficiently. Actually, with DISTSTYLE ALL, the table is replicated to all nodes, but within each node, the data is stored on a single slice (the leader node assigns the table to one slice per node). So queries that scan the entire table may only use one slice per node, leading to underutilization. Option A is wrong because the sort key affects data ordering, not slice utilization. Option C is wrong because WLM queues affect concurrency, not slice usage. Option D is wrong because the query is not using a single node but multiple nodes, but only one slice per node.

Answer

The query is running on the leader node instead of the compute nodes.

Answer

The workload management (WLM) queue is configured with a single query slot.

Answer

The table does not have a sort key defined.

Question 18

A data engineer manages an Amazon Redshift cluster that hosts a 10 TB data warehouse. The cluster uses a single node of type dc2.large (160 GB SSD). The engineer notices that the cluster's disk space is 95% full, and queries are running slowly. The engineer runs the STV_PARTITIONS view and sees that many slices have high 'tossed' counts. The engineer also runs VACUUM and ANALYZE commands, but the disk space does not improve. The engineer suspects that the cluster needs more storage. However, the company wants to minimize cost. Which action should the engineer take to resolve the disk space issue most cost-effectively?

Accepted Answer

Replace the cluster with a single ds2.xlarge node.. Option C is correct because dc2.large is a dense compute node with limited SSD storage; upgrading to ds2.xlarge provides more storage (2 TB HDD) at a lower cost per GB compared to scaling up to dc2.8xlarge. Option A is wrong because adding more dc2.large nodes increases storage but also CPU and memory, which may be unnecessary and more expensive than a single ds2 node. Option B is wrong because dc2.8xlarge has 2.56 TB SSD, which is more expensive than ds2.xlarge. Option D is wrong because switching to RA3 nodes is costly and designed for managed storage, which may be overkill.

Answer

Switch to a ra3.xlplus node with managed storage.

Answer

Scale the cluster to a single dc2.8xlarge node.

Answer

Add another dc2.large node to the cluster to increase total storage.

Question 19

A data engineer is troubleshooting a failed AWS Glue ETL job that reads from Amazon S3 and writes to Amazon Redshift. The job fails with the error: 'ERROR: Cannot insert a duplicate key into unique index'. The Redshift table has a primary key on the 'id' column. The data in S3 contains multiple records with the same 'id'. The engineer needs to ensure that only the latest record for each 'id' is loaded into Redshift. The data has a 'timestamp' column. Which approach should the engineer take?

Accepted Answer

Use the 'dropDuplicates' transformation in the Glue ETL script, ordering by 'timestamp' descending to keep the latest record for each 'id'.. Option B is correct because using the AWS Glue 'dropDuplicates' transformation on the 'id' column, ordering by 'timestamp' descending, will remove duplicate 'id' values, keeping the latest record. Option A is wrong because disabling constraints does not prevent duplicates; it only defers the error. Option C is wrong because setting 'overwrite' mode replaces the entire table, not just duplicates. Option D is wrong because staging tables and MERGE require additional steps and are not directly available in Glue without custom logic.

Answer

Set the write mode to 'overwrite' in the Glue job to replace the entire Redshift table.

Answer

Load data into a staging table in Redshift and then use a MERGE operation to insert only new records.

Answer

Disable primary key constraints on the Redshift table before loading.

Question 20

A company runs a data pipeline on Amazon EMR that processes terabytes of data daily. The pipeline reads from Amazon S3, performs transformations using Spark, and writes results back to S3. Recently, the data engineer noticed that the EMR cluster's spot instances are frequently reclaimed, causing job failures and delays. The cluster uses a mix of On-Demand and Spot instances. The engineer wants to minimize job interruptions while keeping costs low. The current configuration uses a single EMR cluster with a core node group of 10 On-Demand instances and a task node group of 20 Spot instances. The job failures occur during the shuffle phase when tasks on Spot instances are lost. The engineer has no control over when spot instances are reclaimed. Which action will MOST effectively reduce job failures while maintaining cost efficiency?

Accepted Answer

Enable EMR managed scaling to automatically add On-Demand instances when Spot instances are reclaimed.. Option B is correct because enabling managed scaling allows EMR to automatically add On-Demand instances to replace reclaimed Spot instances, ensuring capacity without manual intervention. Option A is incorrect because using all Spot instances increases risk. Option C is incorrect because using all On-Demand increases cost significantly. Option D is incorrect because increasing On-Demand count in core node group may help but does not dynamically adjust; managed scaling is more effective.

Answer

Increase the number of On-Demand instances in the core node group to 20.

Answer

Configure the task node group to use only Spot instances and increase the bid price to the On-Demand price.

Answer

Change the task node group to use only On-Demand instances.