Databricks-GenAI-Assoc Data Preparation — 30 Questions

Question 1

A team is preparing data for a machine learning model and needs to handle missing values in a feature vector. Which TWO techniques are standard practices in Databricks for handling nulls in large-scale feature engineering pipelines?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Apply the fillna() method with a static constant or mean value.. Handling missing data is a critical step in feature preparation to ensure model stability and accuracy. Using native Spark SQL functions like 'coalesce' or the 'fillna' method in PySpark ensures that operations are pushed down to the query optimizer for maximum performance. These techniques prevent null-pointer exceptions during model training and maintain the integrity of feature distributions, which is vital for reproducible machine learning workflows in production environments.

Answer

Use the dropna() method to remove all rows with null values.

Answer

Manually iterate through the DataFrame using a Python loop.

Answer

Convert the entire DataFrame to a Pandas object for imputation.

Question 2

A Data Engineer needs to handle schema evolution in a Delta Lake pipeline where source systems frequently add new columns. Which configuration ensures that the write operation automatically updates the target table schema without manual intervention?

Accepted Answer

spark.databricks.delta.schema.autoMerge.enabled = true. Schema evolution is a powerful feature in Delta Lake that simplifies pipeline maintenance by automatically updating the target table schema when the source DataFrame contains new columns. By setting 'mergeSchema' to true, the write operation dynamically reconciles differences between the source and target. This is critical for data pipelines dealing with streaming sources or legacy systems that do not have rigid, pre-defined schema definitions.

Answer

ALTER TABLE target_table SET TBLPROPERTIES ('schema.evolution' = 'true')

Answer

Use the 'REPLACE WHERE' clause in the write command.

Answer

Configure the table as a Hive-Metastore managed table.

Question 3

When preparing data for a high-frequency streaming dashboard, a Data Engineer finds that the downstream Delta table is being flooded with small files, causing slow reads. Which tool should be used to rectify this without stopping the streaming job?

Accepted Answer

Set 'delta.autoOptimize.optimizeWrite' to true.. Optimized Writes and Auto-Compaction are designed to handle small file issues automatically in Delta Lake. By enabling these features, the Databricks engine automatically coalesces small writes into larger, more efficient files at commit time. This ensures that the table remains performant for read-heavy operations, such as dashboards, without requiring the Data Engineer to run manual vacuum or optimize jobs that might interrupt data availability.

Answer

Run the OPTIMIZE command manually every hour.

Answer

Increase the shuffle partitions to 2000.

Answer

Use the REORG TABLE command.

Question 4

You are migrating a legacy ETL pipeline to Databricks. Which THREE of the following steps are considered best practices for optimizing data preparation in a Delta Lake environment?

Accepted Answer

Apply Z-Ordering on frequently filtered columns.. Optimizing data preparation requires leveraging Delta Lake’s unique features. Implementing partition pruning, Z-Ordering, and using Delta's native caching mechanisms effectively reduces I/O and improves query speeds. These practices are fundamental to ensuring that large-scale datasets remain responsive as they grow, enabling efficient analytics and machine learning workflows. Following these standards ensures compatibility with Unity Catalog and allows for better cost management within Databricks compute resources.

Answer

Partition by columns with very high cardinality.

Answer

Store all data in a single massive file for simplicity.

Question 5

Refer to the exhibit. You are implementing a data quality framework using Delta Live Tables. What happens when a record arrives with a null value in the 'id' column?

Accepted Answer

The pipeline execution stops immediately.. In Delta Live Tables (DLT), quality expectations define how data is validated during the pipeline execution. When 'fail_on_error' is set to true (or the equivalent expectation is violated), the pipeline stops the processing of the batch. This prevents corrupted or incomplete data from reaching the target tables, which is critical for maintaining high-quality downstream datasets in automated data engineering workflows, ensuring that only validated data is used for analysis.

Answer

The null record is dropped, and the pipeline continues.

Answer

The null record is routed to a 'quarantine' table.

Answer

The record is updated to a default value of 0.

Question 6

During data preparation, you observe severe data skew in a join operation. Which strategy is best for mitigating this skew?

Accepted Answer

Use the SKEW hint in the Spark SQL join.. Data skew occurs when one partition contains disproportionately more data than others, slowing down distributed tasks. Skew joins, often handled via 'skew hints' in Spark, redistribute the skewed keys across multiple executors. This prevents a single task from becoming a bottleneck, significantly reducing the overall execution time. It is a critical performance tuning technique for large-scale data preparation where certain keys appear much more frequently than others.

Answer

Increase the shuffle partitions drastically to 10,000.

Answer

Switch to a broadcast join for all large tables.

Answer

Re-partition the data by a random column.

Question 7

Refer to the exhibit. The join fails due to ambiguous column names. How can you resolve this while keeping the 'id' column in the final result?

Accepted Answer

Use df1.join(df2, df1("id") === df2("id")).. When joining two DataFrames with the same column name, Spark's default join operation can cause ambiguity errors if the join key is also present in both tables. By using the 'join' method with a column expression instead of a simple column name string, you can explicitly reference the keys. Alternatively, renaming columns before the join is a standard, robust practice to avoid collisions and maintain clear, readable code in data pipelines.

Answer

Drop the 'id' column from df2 before joining.

Answer

Set 'spark.sql.crossJoin.enabled' to true.

Answer

Cast the 'id' column to a string type.

Question 8

A Data Engineer is using Databricks to prepare data for a machine learning model. The source data has significant imbalance. Which technique is most effective for addressing this imbalance during the preparation stage?

Accepted Answer

Use the sampleBy() method to perform oversampling.. Resampling—specifically undersampling the majority class or oversampling the minority class—is the standard technique to mitigate class imbalance. In Databricks, this is performed efficiently using DataFrame transformations. By rebalancing the dataset, you ensure that the machine learning model does not favor the majority class, leading to better predictive accuracy and fairer model performance across all target categories.

Answer

Increase the learning rate during the training phase.

Answer

Convert the target variable to a continuous value.

Answer

Ignore the imbalance as models handle it naturally.

Question 9

You are preparing a dataset by calculating a rolling average over a window of 30 days. Which Window function specification is required to ensure the average correctly considers only the preceding 30 days?

Accepted Answer

Window.orderBy('date').rowsBetween(-30, 0). Window functions are the standard way to perform time-series analysis in Databricks. Using 'Window.orderBy(col).rowsBetween(-30, 0)' defines a frame that looks at the current row and the 30 preceding rows. This is essential for calculating moving averages correctly in financial or operational data, ensuring that the calculation is temporally accurate and reflects the requested window size without including future or irrelevant historical data points.

Answer

Window.partitionBy('date').orderBy('date')

Answer

Window.partitionBy('id').rangeBetween(0, 30)

Answer

Window.orderBy('date').rowsBetween(0, 30)

Question 10

A data engineer is preparing a dataset for a LLM fine-tuning task. The raw data consists of millions of JSON files stored in Unity Catalog. Which approach provides the highest performance for loading this data into a PyTorch-based training pipeline?

Accepted Answer

Convert the JSON files to a Delta table and use the 'databricks-datasets' library or Petastorm for streaming.. Using Databricks Mosaic AI Model Training or leveraging the Petastorm library with Parquet files optimizes I/O throughput. Converting nested JSON to Delta tables allows for predicate pushdown and efficient partitioning. This matters because inefficient data loading creates a bottleneck in GPU utilization, significantly increasing the cost and duration of fine-tuning runs while potentially causing memory errors during batch processing.

Answer

Read the JSON files directly using a Python loop and convert them to tensors sequentially.

Answer

Convert the raw JSON files into a single massive CSV file on the DBFS root.

Answer

Use a Pandas UDF to iterate through all JSON files on the driver node.

Question 11

Which TWO of the following steps are essential when cleaning unstructured text data for RAG applications within Databricks?

Accepted Answer

Remove sensitive PII using regex patterns and ensure standard normalization.. Cleaning unstructured text involves noise reduction (HTML stripping) and balancing chunk sizes to optimize retrieval. Proper preparation ensures that embeddings capture semantic meaning accurately. If the text contains artifacts or irrelevant metadata, the vector similarity search will return noisy results, degrading the quality of the LLM responses. These steps are foundational to ensuring that the RAG system provides reliable, context-aware information to end-users.

Answer

Apply heavy stemming and stop-word removal to all text chunks.

Answer

Convert all text to lowercase to ensure absolute consistency.

Answer

Replace all special characters with numeric placeholders.

Question 12

Refer to the exhibit. A data engineer is using a JSON-based configuration to prepare log data. Given the policy is 'strict_schema', what happens if an incoming log entry contains a field not defined in the Delta table schema?

Accepted Answer

The write operation fails with an AnalysisException.. Under a 'strict_schema' policy, Databricks will reject any record that deviates from the predefined structure. This is critical for data quality in production pipelines, preventing schema drift. By enforcing strict adherence, the system ensures that downstream LLM applications do not receive malformed data, which could otherwise lead to unexpected parsing errors or inconsistent retrieval results during vector index generation.

Answer

The field is ignored, and the rest of the record is inserted successfully.

Answer

The record is written to a 'dead-letter' queue for manual review.

Answer

The system automatically adds the new field as a column to the table.

Question 13

Which THREE techniques are commonly used to optimize data preparation for embedding generation in Databricks?

Accepted Answer

Using Delta tables to partition data by relevant categories.. Optimizing embedding generation requires balancing compute efficiency with data quality. Techniques like vectorized processing, partitioning, and using pre-computed embeddings help minimize costs and improve speed. Efficiently preparing data ensures that the GPU/CPU resources on the cluster are fully utilized during the vectorization phase, which is often the most time-consuming part of setting up a RAG or semantic search system in a Databricks environment.

Answer

Executing embedding generation as a single-threaded Python process on the driver.

Answer

Avoiding the use of Spark SQL entirely for data cleaning.

Question 14

An engineer notices that a feature vector table in Delta is becoming excessively fragmented after frequent 'upserts'. Which action should be taken to optimize the table for future reads during inference?

Accepted Answer

Run the OPTIMIZE command with ZORDER on the primary key columns.. Frequent upserts in Delta Lake lead to many small files, which degrades read performance. Running 'OPTIMIZE' with 'ZORDER BY' on the frequently queried columns (e.g., entity_id) collocates related information, speeding up retrieval. This is vital for low-latency inference in RAG applications, where the system must fetch vector embeddings quickly to provide real-time responses to user queries, ensuring a smooth user experience.

Answer

Drop the table and re-create it from scratch every week.

Answer

Change the table format to CSV to reduce file management overhead.

Answer

Disable the Delta transaction log to speed up future writes.

Question 15

Refer to the exhibit. You are appending new, streaming JSON logs to an existing Delta table. What is the most appropriate way to handle this schema evolution?

Accepted Answer

Apply a schema-enforcing transformation to cast incoming data to the target types.. Delta Lake supports schema evolution, but incompatible changes (like changing a column type from String to Integer) require manual intervention or explicit casting. The engineer must ensure that the ingestion pipeline aligns with the existing schema. This ensures data consistency for downstream LLM applications, which rely on predictable inputs to perform accurate feature extraction and vector generation without encountering runtime type errors.

Answer

Ignore the error and let the job continue running.

Answer

Enable 'mergeSchema' in the writeStream configuration.

Answer

Delete the entire Delta table and reload from the raw source.

Question 16

An enterprise engineering team is preparing unstructured PDF invoices stored in a Unity Catalog volume for a Retrieval-Augmented Generation pipeline. Which ingestion approach best aligns with Databricks-recommended data preparation practices for generative AI?

Accepted Answer

Ingest the files into a Delta table from the Unity Catalog volume, then use LangChain integration alongside Databricks Vector Search to parse and index the documents.. Utilizing standard PySpark or Auto Loader to load files from the volume into a Delta table, followed by applying vector search indexing, ensures secure governance and scalable ingestion. Unity Catalog provides robust access control for the underlying unstructured files. Managing unstructured data through Delta tables and Unity Catalog maintains enterprise data lineage and simplifies downstream document parsing pipelines.

Answer

Manually copy all PDF files to the local driver node of a standard interactive cluster and execute custom Python scripts to parse them into local CSVs.

Answer

Upload the invoices directly into a legacy Hive metastore table as binary blobs without leveraging Unity Catalog security features.

Answer

Store the PDF binaries inside an unmanaged cloud bucket, bypassing Databricks entirely until the model inference stage requires document retrieval.

Question 17

A data engineer is preparing a large set of policy PDFs stored in a Unity Catalog volume for a RAG application. Before generating embeddings, they must remove page headers, footers, and page numbers that repeat across nearly every page. Which approach is most appropriate for cleaning this text at scale on Databricks?

Accepted Answer

Use the ai_parse_document function to extract text and layout from each PDF, then apply rule-based filtering to remove recurring header and footer patterns.. The most efficient and accurate method is to use ai_parse_document to extract text with layout information, then apply rule-based filtering to remove repeating headers and footers. This leverages Databricks-native capabilities and scales across many documents, ensuring clean text for embedding generation.

Answer

Use the binaryFile data source to read raw bytes and apply a regular expression to strip everything before the first newline and after the last newline.

Answer

Convert each PDF page to an image, run OCR with a third-party library, and manually delete header and footer regions using bounding box coordinates.

Answer

Load the PDFs with spark.read.format("pdf") and drop the first and last line of each page based on line index.

Question 18

A GenAI team is preparing a fine-tuning dataset stored as a Delta table in Unity Catalog. They need to ensure that each training example is a valid JSON string conforming to a specific schema before feeding it to the model. They want to validate and optionally quarantine malformed records in a single pass. Which Databricks SQL function is most appropriate for this task?

Accepted Answer

from_json. The from_json function parses a JSON string into a struct using a specified schema, and returns null if the string does not conform. By applying from_json and filtering on null, the team can validate and quarantine malformed records in one pass, ensuring only valid examples are used for fine-tuning.

Answer

schema_of_json

Answer

json_tuple

Answer

to_json

Question 19

A data engineer is preparing a large corpus of text for a RAG application. The text is stored in a Delta table with columns doc_id and content. The engineer needs to split the content into chunks of approximately 512 tokens with a 50-token overlap, and then compute embeddings for each chunk. They want to use Databricks-native capabilities to minimize data movement and cost. Which two approaches are valid and efficient for this task? (Choose two.)

Accepted Answer

Use Spark NLP's DocumentAssembler and Tokenizer to split text into chunks, then use a pre-trained SentenceTransformer within a Pandas UDF to compute embeddings.. Using Databricks Foundation Model APIs or Spark NLP with Pandas UDFs allows distributed, in-cluster processing of both chunking and embedding. These methods avoid data movement and leverage Spark's scalability, making them efficient for large RAG corpora.

Answer

Use the Databricks Vector Search API to directly index the raw text without chunking, relying on its built-in chunking feature.

Answer

Export the text to a Parquet file on cloud storage, then use a separate Python script with the Hugging Face transformers library on a single node to compute embeddings.

Answer

Use a Scala UDF that calls an external embedding API for each row, then write the results to a Delta table.

Question 20

A data engineer is preparing a dataset for training a generative AI model. The dataset contains a column 'text' with raw user reviews. Before tokenization, the engineer needs to remove HTML tags and special characters, convert text to lowercase, and expand common contractions. Which Databricks feature or approach is best suited for this text normalization step?

Accepted Answer

Use Spark NLP's DocumentNormalizer and Normalizer annotators in a Spark ML pipeline.. Spark NLP's DocumentNormalizer and Normalizer are designed for exactly these text cleaning tasks, providing scalable, distributed processing within Databricks. They handle HTML tag removal, lowercasing, and special character removal, and can be customized for contraction expansion.

Answer

Use the ai_query function with a prompt to ask an LLM to normalize the text.

Answer

Use Delta Live Tables expectations to enforce that text is normalized.

Answer

Use the Databricks Feature Store to apply normalization functions during feature computation.