Select Two (Multi-Select) Questions

Use the 'Cluster Logs' to compare the OS kernel versions of the nodes.

Why it fails: Comparing OS kernel versions is rarely the solution for job failures, as Databricks manages the underlying runtime environment. If there is a disparity, it would be due to selecting different Databricks Runtime versions, not the specific OS kernel, which is abstracted away from the end user.

Enable the 'Debug Mode' on the Spark Driver to see raw system calls.

Why it fails: Debug mode for raw system calls is not a standard feature in Databricks and would not provide useful context for comparing environment configurations. This level of debugging is excessively low-level and does not address the high-level configuration differences that typically cause environment-specific pipeline failures.

Query the 'workspace_users' table to see who ran the job last.

Why it fails: Knowing who ran a job last does not help identify configuration differences. While auditing is useful for security, the technical investigation of why a job fails requires comparing settings, libraries, and code, not the identity of the user who triggered the last execution.

Convert the table to a Parquet external table and rely on the query engine to infer partitioning

Why it fails: Moving to a plain Parquet external table discards Delta Lake features such as transaction logs, data skipping statistics, OPTIMIZE, and Z-ORDER or liquid clustering. Without those, the engine cannot benefit from Delta's file-level min/max statistics, so bytes scanned would likely increase rather than decrease for the date-filtered and user-aggregated queries.

Enable Change Data Feed on the table so late-arriving corrections are captured automatically

Why it fails: Change Data Feed records row-level changes for downstream consumers to read incrementally; it does not itself improve scan efficiency or join performance for analytical queries. Enabling it adds storage and write overhead, and it does not address file layout or pruning, so it fails to satisfy the goal of reducing bytes scanned for the date-filtered aggregations.

Partition the table by user_id to maximize file skipping for user-level aggregations

Why it fails: Partitioning by a high-cardinality column such as user_id produces an enormous number of small directories and files, which overwhelms the metastore and slows listing and planning. It also does nothing for the common event_date filter, so queries scanning a date range would still touch most partitions, making this choice counterproductive for the stated workload.

Table ACLs that grant SELECT only on specific columns.

Why it fails: Unity Catalog does not support column-level grants in the GRANT statement. While you can grant SELECT on a table, you cannot restrict which columns a user sees via table ACLs. Column-level security is achieved through column masks or by exposing a view with a subset of columns, so this option does not meet the masking requirement.

Workspace-level IP access list to restrict access to the table.

Why it fails: IP access lists control network origins for workspace access, not row or column visibility within a table. They cannot mask a column or filter rows based on a user's region. This feature addresses a different security concern and does not satisfy the fine-grained data access requirements described.

Dynamic view that joins the table with a mapping table of user regions.

Why it fails: A dynamic view can implement row-level filtering and column masking, but it is a separate object that users query instead of the base table. The scenario asks for fine-grained access control on the Delta table itself, and using a view would not apply the policies directly to the table. Unity Catalog's native row filters and column masks are the intended features here.

Configure single-node clusters with maximum autoscaling limits to handle unpredictable concurrency peaks without cluster management overhead.

Why it fails: Single-node clusters completely lack distributed processing capabilities, restricting query execution to a single driver and causing immediate out-of-memory failures on large datasets. They cannot scale out workloads across multiple worker nodes to handle concurrency effectively.

Provision pools of pre-warmed idle driver nodes to ensure zero-second startup latency for all analytical queries.

Why it fails: Pre-warmed idle driver nodes do not address concurrency spikes, since workers execute queries, and idle drivers waste cost. It is tempting because eliminating cold starts sounds attractive, and would suit predictable latency-sensitive workloads, not sporadic unpredictable analytics needing elastic worker scaling.

Disable automatic cluster termination and keep all worker nodes running 24/7 to guarantee immediate resource availability.

Why it fails: Disabling auto-termination causes clusters to run continuously, racking up massive idle charges during off-peak hours and weekends. This is an anti-pattern for cost optimization in cloud environments where resources should scale down when not in use.

Users can directly mount external locations as DBFS paths.

Why it fails: Unity Catalog's external location model is intended to replace or augment traditional DBFS mounts. While it provides similar functionality, it does not rely on global DBFS mounts. Instead, it provides granular, policy-driven access to data files, which is a much more secure and manageable approach for enterprise data governance.

External locations are automatically created for every S3 bucket in the account.

Why it fails: Unity Catalog does not automatically import or create external locations for all S3 buckets. This would be a security nightmare, as it would expose all storage resources by default. Administrators must explicitly register each location, ensuring that only necessary data assets are governed and made accessible within the platform.

External locations are only supported for Delta-formatted data.

Why it fails: Unity Catalog supports various data formats in external locations, not just Delta. While Delta is the recommended format for performance and governance, Unity Catalog is designed to be flexible, allowing users to register and govern external tables in formats like Parquet, CSV, or JSON for compatibility with legacy systems.

Executing the 'run_job' command inside a notebook cell.

Why it fails: There is no built-in `run_job` command within the notebook environment to trigger an external job definition. Notebooks are execution environments, and while they can be used to run jobs, they do not have a native, top-level command to trigger another job directly by name.

Running the 'dbutils.jobs.run()' method.

Why it fails: The `dbutils` library does not have a `jobs` module that exposes a `run()` method for triggering jobs. While `dbutils` is a powerful tool for file system and secret management, it is not the interface for orchestrating job execution within the workspace.

Creating a file named 'trigger.job' in DBFS.

Why it fails: Creating files in DBFS does not trigger any background jobs in Databricks. Databricks job execution is managed through the platform's orchestration service, which operates independently of file system contents, making this an ineffective way to automate or manage job runs.

Increase the number of shuffle partitions to 2000.

Why it fails: Increasing shuffle partitions can help with large data volumes but may create many small partitions, increasing overhead. It does not directly address skew; skewed keys will still cause uneven partition sizes. Without AQE or salting, this may not resolve the issue and could worsen performance due to task scheduling overhead. It is not a targeted solution for skew.

Cache the larger table in memory before the join.

Why it fails: Caching a large table may consume significant memory and cause spilling, potentially slowing down the job. It does not address the shuffle or skew issues. Caching is beneficial for iterative algorithms or repeated access, but for a single join, it may not provide a net benefit and could increase cost by requiring a larger cluster. It is not the most effective optimization here.

Repartition the larger table on the join key before the join.

Why it fails: Repartitioning the larger table on the join key can help with skew but also introduces a full shuffle, which is expensive. If the smaller table is broadcast, no shuffle is needed at all. Repartitioning is often used to mitigate skew, but it should be combined with other techniques like salting. In this scenario, broadcasting is more effective and avoids the shuffle entirely.

Configure a Databricks SQL warehouse for the recipient to query the shared data.

Why it fails: The recipient is using a non-Databricks client, so they do not need a Databricks SQL warehouse. Delta Sharing provides a REST API endpoint that the recipient's client connects to directly. A SQL warehouse is only needed if the recipient were using Databricks SQL to query the shared data. Thus, this action is unnecessary and would not facilitate access for the external client.

Enable the Delta Sharing server on the Databricks workspace.

Why it fails: Delta Sharing is enabled at the metastore level, not per workspace, and it is typically enabled by default in Unity Catalog. There is no separate server to enable. The sharing is managed through Unity Catalog, and the recipient accesses the data via the metastore's Delta Sharing endpoint. Thus, this step is not required and is not a valid configuration action.

Grant the recipient SELECT privileges on the shared table.

Why it fails: Delta Sharing does not use Unity Catalog privilege grants like SELECT on tables for recipients. Access is controlled by adding the table to a share and granting the recipient access to that share. The recipient's permissions are managed at the share level, not through standard SQL grants. Therefore, this action is not required and would not enable access.

Use dynamic views to mask PII columns for all users except administrators.

Why it fails: While dynamic views can mask PII, they require manual creation and maintenance for each table, which is less scalable than tag-based ABAC policies. They also do not inherently provide auditing. The requirement for auditable access is better met by audit logging, and for sensitive data protection, ABAC with tags is more centralized and less error-prone.

Create a separate metastore for each department to isolate data.

Why it fails: Creating separate metastores per department increases complexity and prevents centralized governance and auditing. Unity Catalog is designed to provide a unified governance layer across the organization. Isolating metastores would fragment audit logs and make it harder to enforce consistent policies, thus not meeting the requirements for auditable and protected data access.

Store all data in a single catalog and grant `ALL PRIVILEGES` to the data engineering team.

Why it fails: Granting `ALL PRIVILEGES` to a broad group violates the principle of least privilege and does not protect sensitive data. This approach would make auditing more difficult because many users would have unrestricted access. It does not meet the requirement for protecting sensitive data and could lead to unauthorized access.

Hardcode credentials directly into the notebook for ease of access.

Why it fails: Hardcoding credentials poses a severe security risk, as sensitive keys are exposed in version control and logs. Credentials should always be managed using Databricks Secrets and referenced programmatically, ensuring that access tokens remain encrypted and restricted to authorized users only, maintaining compliance with security standards.

Avoid using libraries and rely only on built-in Spark functions.

Why it fails: Restricting code to built-in functions limits the functionality of the pipeline and prevents the use of proven, specialized libraries. Proper dependency management, such as defining requirements via cluster libraries or environment configuration, is a standard and recommended practice for extending Spark capabilities in a reproducible manner.

Deploy code directly from the workspace to production.

Why it fails: Deploying directly from the workspace without a proper CI/CD pipeline or version control process is dangerous. It lacks auditability and increases the likelihood of human error. Automated deployments via CI/CD pipelines ensure consistency and quality by running tests before code is ever moved to the production environment.

Automatic indexing for all columns

Why it fails: Delta Lake does not automatically index every column in the dataset. While it supports Z-Ordering and data skipping, these are opt-in features that the engineer must configure. Assuming automatic indexing is a misconception that can lead to performance issues if queries are not properly optimized using the correct clustering and partitioning strategies.

Built-in support for real-time web socket streaming

Why it fails: Delta Lake does not handle web socket streaming; it is a storage layer for structured data. While it integrates with Structured Streaming for incremental data processing, it is not a networking or communication tool. Confusing storage capabilities with communication protocols is a common error when planning the architecture of real-time data ingestion systems.

Apply business logic and complex transformations to create aggregate summary tables.

Why it fails: Aggregation should generally happen at the Gold layer. The Silver layer is intended to be a refined, tabular version of the source data, kept at the grain of the input. Creating summaries at the Silver layer reduces the granularity and flexibility of the data, making it less useful for general analysis.

Convert all column names to uppercase to ensure case-insensitive consistency.

Why it fails: While standardizing column names is a good practice, it is not a 'critical' step for data quality in the same way as deduplication or type enforcement. It is an optional naming convention rather than a fundamental requirement for ensuring data integrity or structural validity in a medallion architecture.

Increase the number of partitions to the maximum possible value to ensure maximum parallelism.

Why it fails: Setting partitions to the maximum value can cause 'too many small tasks' overhead, where the time to schedule and manage tasks exceeds the time to actually process the data. Efficient partitioning requires balancing parallelism with the amount of data processed per task to avoid unnecessary cluster management overhead.

Use Cross Join for every join operation to ensure no data rows are missed.

Why it fails: Cross joins create a Cartesian product of all rows, which is computationally explosive and usually unintended. It results in massive data volumes that can cause OOM errors and infinite execution times. It is the least efficient way to perform a join and should be strictly avoided for large datasets.

Enable multi-cluster load balancing to ensure all queries are executed on the largest instance type.

Why it fails: Multi-cluster load balancing is designed to increase concurrency by spinning up additional clusters, not to force execution on the largest instance type. Using the largest instance type unnecessarily increases costs and does not guarantee improved performance for smaller queries that might be better served by smaller, cost-effective clusters.

Configure SQL warehouse scaling to use the maximum cluster size at all times to avoid resizing overhead.

Why it fails: Setting a warehouse to the maximum size at all times forces the highest billing rate regardless of current workload demands. This is a poor cost-optimization strategy, as it ignores the elasticity benefits of Databricks SQL, leading to significant over-provisioning and wasted budget during periods of low or moderate activity.

Disable the query cache to ensure all results are freshly computed for accurate billing metrics.

Why it fails: Disabling the query cache forces the warehouse to recompute results for every query, which significantly increases compute usage and costs. The query cache is a performance feature designed to return results instantly for identical queries, saving both time and money by avoiding unnecessary re-execution of complex operations.

Set the task timeout to zero to prevent the workflow from ever stopping during a failure.

Why it fails: Setting a timeout to zero is not a valid configuration for ensuring data integrity during dependencies. It essentially disables the monitoring of task execution time, which can lead to hung jobs consuming resources indefinitely without triggering the necessary alerts or failure handling logic required for production stability.

Hardcode the file path of the upstream output into the downstream task configuration.

Why it fails: Hardcoding paths is an anti-pattern that couples tasks tightly and creates maintenance challenges. It ignores the workflow dependency graph, making it impossible for the scheduler to manage retries or state correctly. If the upstream task fails, the downstream task might still try to read non-existent data.

Disable all retries to ensure that the error log is captured immediately upon failure.

Why it fails: Disabling retries does not assist in managing dependencies; it only guarantees the job stops at the first sign of trouble. While it might assist in capturing logs, it fails to utilize Databricks' built-in resilience features that handle transient errors automatically, thus reducing overall workflow reliability and efficiency.

Assign table ownership to individual users for better tracking.

Why it fails: Assigning ownership to individual users creates a significant security risk when those users leave the organization. If an account is deleted, ownership management becomes complex. Best practice dictates using service principals or dedicated groups to manage ownership to ensure continuity and prevent unauthorized access gaps.

Ensure that the metastore admin has access to all data.

Why it fails: The metastore admin role should be treated with extreme caution and follow the principle of least privilege. While they manage metadata, they should not necessarily have unrestricted access to read sensitive business data. Separating administrative duties from data access duties is a critical component of security compliance.

Public access should be granted to the root catalog.

Why it fails: Granting public access to the root catalog is a severe security vulnerability. It allows any user in the workspace to discover and potentially access data objects. Access control should be strictly restricted to specific users and groups using the principle of least privilege to ensure data security.

Grant all developers full cluster permissions to access logs directly on the nodes.

Why it fails: Granting excessive permissions violates the principle of least privilege and introduces security risks. Production environments should use centralized logging services rather than giving individual developers direct access to cluster nodes. Debugging should be performed via standard interface tools, not by logging into underlying infrastructure components directly.

Run all production jobs as the root user to avoid permission-related errors.

Why it fails: Running jobs as the root user is a critical security vulnerability and is not a debugging strategy. Errors related to permissions should be resolved by configuring appropriate IAM roles or service principals, not by bypassing the security model, which exposes the entire workspace to unauthorized file modifications.

It can be used to perform deletes on the target table by issuing manual DELETE commands.

Why it fails: APPLY CHANGES INTO does not support manual DELETE commands within the DLT pipeline logic. The target table should be treated as managed by the DLT engine. If data removal is required, it must be handled through the logic defined within the APPLY CHANGES block using appropriate flags.

It allows for multiple primary keys, but only one can be used for sequencing the updates.

Why it fails: The KEYS clause in APPLY CHANGES INTO expects a defined set of columns that uniquely identify the record. While it supports composite keys, it does not distinguish between 'primary' and 'sequencing' keys in the manner described. Sequencing is handled by a separate SEQUENCE BY clause for timestamp-based ordering.

It automatically creates a history table for SCD Type 1 processing by default.

Why it fails: SCD Type 1 processing performs an 'overwrite' of existing records, meaning history is not preserved. History tables are specifically a feature of SCD Type 2 implementations, where the history is tracked via additional columns like valid_from and valid_to, rather than being an automatic default for all types.

Increase the number of shuffle partitions significantly for small datasets.

Why it fails: Increasing shuffle partitions for small datasets creates unnecessary task overhead. The time taken to schedule and manage thousands of tasks will far outweigh the execution time, often leading to slower performance. Proper partition sizing should align with the actual volume of data being processed in the job.

Always use 'repartition()' before any operation to ensure even data distribution.

Why it fails: Calling repartition() triggers a full shuffle of the data across the cluster. If the data is already well-distributed, this is a massive performance waste. It should only be used when necessary to fix severe data skew, as it is one of the most expensive operations in Spark.

Assign the Service Principal the 'Owner' role on the entire workspace.

Why it fails: Assigning the 'Owner' role violates the principle of least privilege. Service Principals should be granted only the specific permissions needed for their tasks, such as 'USE CATALOG' or 'SELECT' on specific tables. Global ownership provides excessive privileges that increase the risk of accidental or malicious data modification.

Enable multi-factor authentication (MFA) for the Service Principal.

Why it fails: Service Principals do not support interactive MFA because they are machine identities, not human users. Authentication is handled via tokens or secret keys. Requiring human-style MFA for a machine identity is technically impossible and would break any automated process or job relying on that identity for operation.