Refer to the Exhibit Practice Questions

The Databricks cluster needs to be restarted to apply the new IAM policy.

Why it fails: While policy changes can take time to propagate, restarting the cluster is not a requirement for IAM changes to take effect. If the policy were correct, the cluster would eventually assume the new permissions. The error indicates a fundamental lack of permissions in the current policy, not a latency issue.

The S3 bucket policy is blocking the request despite the IAM policy.

Why it fails: While this is a plausible scenario in AWS, the most direct cause based on the provided exhibit is the IAM policy itself. The question asks why the *job* fails, and the provided policy is clearly insufficient for a write operation, regardless of the bucket policy's state.

The 'Resource' ARN is missing the suffix '/*'.

Why it fails: The exhibit explicitly includes '/*' in the Resource field. This allows the permission to apply to objects inside the bucket. The issue is not the resource definition but the lack of the write action. The syntax of the ARN is correct, but the permission scope is too narrow.

The max_capacity limit is set too low, forcing teams to provision multiple competing instance pools.

Why it fails: A max_capacity of 10 limits the maximum number of instances the pool can hold. While restrictive for large teams, it caps rather than increases infrastructure spending and does not explain unexpected continuous baseline charges during idle times.

Instance pools do not support Spot instances, forcing all pool-backed clusters to run expensive on-demand VMs.

Why it fails: Databricks instance pools fully support spot instances when configured correctly. The unexpected cost is not caused by an inability to use spot pricing, but rather by the resource allocation strategy maintaining active idle virtual machines.

The selected node_type_id is optimized for storage rather than compute, causing inflated licensing surcharges.

Why it fails: Instance types like i3.xlarge are storage-optimized instances commonly used for caching and shuffle-heavy workloads. While they have specific pricing structures, storage optimization does not inherently trigger unexpected persistent idle billing compared to minimum idle node counts.

Decrease the number of partitions using spark.sql.shuffle.partitions.

Why it fails: Reducing shuffle partitions results in larger individual data blocks per task. If the cluster is already struggling with memory limits, larger partitions will exacerbate the OOM issue, leading to more frequent executor failures rather than resolving them. This setting should be increased to handle large datasets effectively.

Enable autoscaling to add more nodes to the cluster.

Why it fails: Adding more nodes increases total cluster compute capacity but does not increase the amount of memory available to a single executor container. If the issue is a single task exceeding the container limit, distributing more tasks across additional nodes will not prevent that specific container from failing.

Change the file format from Parquet to CSV for better performance.

Why it fails: Parquet is a highly optimized columnar format that is significantly more memory-efficient than CSV. Switching to CSV would increase the data footprint in memory and likely lead to even worse memory pressure and performance degradation. It is not a valid strategy for resolving memory-related OOM errors.

SELECT COUNT(*) FROM my_table WHERE updated_at < now() - interval 1 hour

Why it fails: This query returns the count of old records, not whether the most recent data is stale. If the table has many historical records, this query will return a high value even if new data is being ingested perfectly, leading to false-positive alerts.

SELECT updated_at FROM my_table ORDER BY updated_at DESC LIMIT 1

Why it fails: This query returns the timestamp itself, not a boolean flag or a metric that can be easily compared to a constant in a standard threshold-based alert configuration, making it difficult to trigger a boolean 'Alert' state directly.

SELECT datediff(now(), max(updated_at)) FROM my_table

Why it fails: While this returns the difference in days, it is not ideal for hourly thresholds. Even with proper conversion, the result is a number representing a duration, and if the data is fresh, the result might be zero, which is harder to alert on than a boolean flag.

The table is locked by another process.

Why it fails: Locking issues manifest as concurrency errors or timeouts, not as an AnalysisException related to data type incompatibility. The error clearly indicates a schema validation failure, which happens during the initial write process, confirming that the issue is related to the data structure, not the concurrency state of the table.

The table schema is corrupted and needs to be repaired.

Why it fails: Schema corruption errors are much rarer and typically involve the transaction log itself. This error is a clear semantic mismatch between the source and target schema definitions. The system is functioning correctly by preventing the insertion of data that does not fit the defined schema of the target table.

The user does not have write access to the table.

Why it fails: Insufficient permissions would result in an AccessDenied or Authorization error. The error message is a technical validation error from the Spark SQL engine, indicating that the schema enforcement process has identified an issue with the data being submitted, which is completely separate from the user's authorization level.

When the query returns a row count greater than 3600.

Why it fails: The configuration explicitly targets the 'duration' column as the evaluation metric for the alert condition. The row count of the result set is irrelevant to this specific trigger configuration, which focuses exclusively on the time-based performance of the query execution.

When the query execution takes longer than 3600 seconds to run.

Why it fails: The alert is based on a specific column in the query results, not the internal query runtime metadata. While the query might return a value representing duration, the alert evaluates the data returned by the query, not the Databricks SQL execution engine's internal timer.

When the query fails and returns an error code equal to 3600.

Why it fails: Alerts in Databricks SQL are based on successful query result sets that meet a condition, not on trapping application or SQL errors. If a query fails, the alert cannot process the output, and the state will remain in the last known valid configuration.

The user is allowed to modify the alert's query criteria and notification frequency.

Why it fails: The 'CAN_VIEW' permission only grants read-only access to the alert. It does not provide the 'CAN_MANAGE' or 'CAN_EDIT' permissions required to modify the underlying SQL query, threshold logic, or notification settings. Therefore, the user cannot change the operational behavior of this specific SQL Alert configuration.

The user has sufficient permissions to delete the alert from the Databricks SQL workspace.

Why it fails: Deleting an object in Databricks requires the 'CAN_MANAGE' permission level. 'CAN_VIEW' is the lowest permission tier in the access control hierarchy and does not include destructive capabilities. This policy structure is designed to enforce the principle of least privilege, preventing users from accidentally or intentionally removing critical system monitors.

This JSON policy provides the user with global access to all alerts within the SQL warehouse.

Why it fails: The JSON policy specifically targets the resource '/alerts/12345', which refers to a single, unique alert object. It does not grant broader access to other alerts, folders, or the SQL warehouse itself. Scoped access control is a fundamental security practice in Databricks to prevent unauthorized exposure of enterprise data monitoring tools.

The engine silently ignores the invalid rows and completes the merge for valid records only.

Why it fails: Delta Lake prioritizes data integrity and does not perform silent drops by default. Any violation of a table constraint results in the failure of the atomic transaction. This prevents the silent loss of data and forces engineers to address the underlying quality issue rather than allowing corrupt data to propagate.

The invalid rows are automatically routed to a hidden sidecar table for later review.

Why it fails: Delta Lake does not automatically create sidecar tables for rejected rows during standard Merge operations. While Delta Live Tables supports this via 'expectations' with 'fail' or 'drop' policies, standard SQL merge operations require explicit handling of invalid data or will simply fail the entire transaction if constraints are violated.

The engine automatically updates the invalid rows to the default column value.

Why it fails: Delta Lake does not modify incoming data to satisfy constraints automatically. Attempting to insert a null value into a column marked as NOT NULL will trigger a runtime exception. The engine assumes the input data must be cleaned before ingestion or that the logic should be handled by the application code.

Run a DELETE statement for the partition and then an APPEND.

Why it fails: This approach is two separate transactions, which can lead to data inconsistency if a reader accesses the table in between. It is also less performant than a single atomic overwrite operation. Using partition overwrite is a more reliable and efficient way to achieve the same result in one step.

Use the MERGE command to overwrite only the records in the specific partition.

Why it fails: While MERGE can perform updates, it is not the standard or most efficient way to replace an entire partition. MERGE is designed for row-level upserts. Partition overwrite is optimized for the bulk replacement of data, which is faster when the entire contents of a partition are being replaced.

Set 'spark.databricks.delta.retentionDurationCheck.enabled' to false.

Why it fails: This setting is for managing the retention period of deleted files during VACUUM operations. It has nothing to do with overwriting partitions or controlling the behavior of the write operation. This configuration change would not help with replacing a specific partition and is irrelevant to the requirement presented.

Add 'cloudFiles.schemaEvolutionMode': 'rescue'.

Why it fails: The 'rescue' mode is used to capture data that does not match the inferred schema into a special column, not to evolve the schema itself. It helps prevent data loss for malformed records but does not automatically update the table schema to include the new columns found in source files.

Add 'cloudFiles.maxFiles': '1000'.

Why it fails: This parameter controls the maximum number of files to process per trigger, which affects ingestion throughput and latency. It has absolutely no bearing on schema evolution or the detection of new data fields within the source files. Increasing this will not resolve the failure to capture new columns.

Add 'cloudFiles.allowOverwrites': 'true'.

Why it fails: Allowing overwrites is used to handle situations where files might be re-uploaded to the same location, which is generally discouraged in streaming pipelines. It does not control schema detection logic or the propagation of new field names from source JSON files into the existing Delta table schema.

Write latency will increase, but read performance will remain unaffected.

Why it fails: While there is a slight performance overhead during the write process to reorganize data, the primary goal is improving read performance. Read performance will definitely be improved by these settings, as they prevent the creation of small files that would otherwise slow down subsequent read operations across the table.

Storage costs will increase significantly due to the creation of temporary duplicate data files.

Why it fails: These settings do not create permanent duplicate files. While compaction involves rewriting data, Delta Lake's vacuuming process eventually removes the old files. The minor temporary storage increase is negligible compared to the massive performance benefits gained by having an efficiently partitioned and compacted table layout for analytical read tasks.

The cluster will fail to start because these configurations are deprecated in recent Databricks runtimes.

Why it fails: These settings remain fully supported and are considered best practices for managing small files in Delta Lake. They are not deprecated, and the cluster will start successfully. The use of these flags is a standard recommendation for ensuring long-term table performance without relying solely on manual maintenance tasks.

The partition columns should include 'customer_id' to improve the pruning speed.

Why it fails: Adding high-cardinality columns like 'customer_id' to the partition strategy leads to the small file problem. This increases metadata overhead and creates thousands of unnecessary directories, which actually slows down query performance and makes file management significantly more difficult to automate within the Databricks environment.

Z-Ordering must be performed on the partition columns instead of the join columns.

Why it fails: Z-Ordering is meant for non-partition columns, typically those with high cardinality used in filters or joins. Z-Ordering on partition columns is redundant because partition pruning already handles the filtering logic. The issue here is the lack of partition pruning in the query, not the Z-Ordering configuration itself.

The file format should be changed to Parquet to improve individual file read performance.

Why it fails: Delta Lake is built on top of Parquet and provides features like Z-Ordering and partition pruning that raw Parquet does not support natively. Changing the format to Parquet would remove the ability to use Delta-specific optimizations, making the query performance even worse while losing critical ACID transaction capabilities.

The cluster is running an outdated version of the Spark runtime.

Why it fails: An outdated runtime would typically trigger incompatibility errors or library missing errors rather than a table-not-found exception. The error clearly specifies a metadata lookup failure within the catalog service, which is independent of the specific Spark version running on the cluster nodes during job execution.

The job is running under a service principal that lacks permissions to the Hive Metastore.

Why it fails: Insufficient permissions usually result in an 'AccessDenied' or 'AuthorizationException' rather than a 'TableNotFound' error. The system is successfully querying the catalog but failing to locate the specific table object, implying a configuration or environment mismatch rather than a security or identity-based restriction issue.

The cluster has insufficient memory to load the table metadata.

Why it fails: Metadata operations are lightweight and managed by the metastore service, not the cluster's RAM. Insufficient memory would manifest as a 'java.lang.OutOfMemoryError' during the processing stage, not as a failure to resolve a table name. The error is strictly related to catalog resolution and object location.

Task D reached its execution timeout limit before it could start.

Why it fails: A timeout would result in a 'Failed' status, not a 'Skipped' status. 'Skipped' indicates that the scheduler decided not to run the task, not that it attempted to run and failed due to a duration limit. The dependency failure is the logical explanation for the skipped status.

The cluster running Task D was terminated by an administrator.

Why it fails: Terminating a cluster would lead to a 'Failed' status for all affected tasks. 'Skipped' is a specific status indicating that the task logic determined the execution was unnecessary or blocked by dependency constraints. It is not a status associated with infrastructure or resource-level interruptions during the job run.

Task D was manually disabled in the job configuration.

Why it fails: While it is possible to disable tasks, the exhibit implies that Task D was expected to run as part of the sequence. If it were disabled in the configuration, it would not appear in the execution history as 'Skipped' in the context of a dependency graph failure.

Grant 'SELECT' on the 'q1_data' table to the user.

Why it fails: Granting 'SELECT' is insufficient because the user lacks the 'USE CATALOG' privilege. The security model requires the user to have traversal rights on all parent objects, including the catalog and schema. Without these rights, the SQL engine will fail to resolve the path to the table regardless of table-level access.

Change the user's role to 'Account Admin' to bypass restrictions.

Why it fails: Assigning 'Account Admin' is a massive security violation. Admin roles provide broad access that goes far beyond what is needed to read a specific table. Permissions should always be assigned based on the principle of least privilege, providing exactly the access required for the user's specific business function.

Move the 'q1_data' table to a catalog where the user has access.

Why it fails: Moving data to bypass security restrictions is a poor practice that leads to fragmented data management and inconsistent governance. If the user needs access to the data, the correct approach is to grant them the appropriate permissions on the existing catalog, not to move the data itself.