Databricks-GenAI-Assoc Data Preparation Practice Test 3 — 15 Questions

Question 1

Which Databricks feature is best suited for maintaining the lineage of data used during the preparation of training sets for Generative AI?

Accepted Answer

Unity Catalog. Unity Catalog provides centralized governance, including automated data lineage tracking. This allows engineers to trace the source of data used for model training, which is vital for reproducibility and regulatory compliance. Knowing the exact provenance of training data helps in troubleshooting model bias and ensures that sensitive data sources are correctly identified and managed throughout the lifecycle of the model training process.

Answer

Delta Lake Time Travel

Answer

Cluster Policies

Answer

MLflow Experiments

Question 2

When preparing a dataset for fine-tuning an LLM, you need to ensure the data is representative of the target domain. What is the most effective approach to detect and mitigate sampling bias in your training set using Databricks?

Accepted Answer

Perform statistical profiling on feature distributions and re-sample if necessary.. Using exploratory data analysis (EDA) with Databricks SQL or Spark to analyze distribution statistics is the most effective way to identify bias. By comparing the distribution of the training set to a representative sample of real-world production data, engineers can identify under-represented categories. This ensures that the fine-tuned model performs reliably across all expected input scenarios, preventing the model from developing blind spots.

Answer

Increase the batch size during the fine-tuning process.

Answer

Change the model architecture to a larger parameter count.

Answer

Use a random seed to shuffle the data before splitting.

Question 3

Which of the following describes the purpose of using a 'Feature Store' when preparing data for Generative AI applications?

Accepted Answer

To provide a unified interface for consistent feature computation during training and inference.. The Feature Store acts as a centralized repository for standardized, reusable features. It ensures that the same logic is used for data preparation during both training and inference. This consistency is critical for preventing training-serving skew, where the model's performance in production differs from its training performance because the input data was transformed differently, ensuring the model remains accurate and reliable in real-world scenarios.

Answer

To store raw text files for long-term archival.

Answer

To serve as a high-performance database for LLM model weights.

Answer

To replace the need for data cleaning and preprocessing entirely.

Question 4

You are preparing a large dataset for fine-tuning a model using Databricks Delta Live Tables (DLT). Which configuration is best for ensuring data quality and lineage in this pipeline?

Accepted Answer

Implement DLT Expectations to flag or drop invalid records.. Delta Live Tables (DLT) with Expectations allows you to define constraints on data quality. By tagging data that fails these constraints, engineers ensure only clean data reaches the final training table. This provides built-in lineage and automatic quality monitoring. In the context of GenAI, clean data is paramount, as noise in the training set leads to poor model performance and unpredictable behavior in downstream applications.

Answer

Use standard Spark SQL jobs without DLT.

Answer

Write all data to a temporary folder and clean it after training.

Answer

Rely on the LLM to automatically filter out low-quality data during training.

Question 5

Which TWO factors are most important when selecting a chunking strategy for text data prior to vectorization?

Accepted Answer

The maximum input token length of the target embedding model.. Selecting a chunking strategy requires balancing semantic context and technical limitations like context window size. If chunks are too small, they lack meaning; if too large, they exceed the limits of the embedding model and introduce irrelevant noise. Proper chunking is vital for maximizing the accuracy of RAG systems, as it determines the granularity of the information available for the retriever to present to the LLM.

Answer

The file format of the source documents (e.g., PDF vs. Word).

Answer

The total number of documents in the corpus.

Answer

The storage cost of the resulting vector database.

Question 6

A team is preparing data for a RAG system and needs to remove duplicates from a large collection of PDF text extracts. What is the most efficient way to perform de-duplication in Databricks?

Accepted Answer

Execute a distributed 'dropDuplicates' operation in Spark.. Using Spark's 'dropDuplicates()' on the text content column or calculating a hash (e.g., MD5) of the text to identify duplicates is highly scalable. This is crucial because redundant information in the vector database can cause the retriever to prioritize duplicate entries, leading to biased results and inefficiency. Removing duplicates ensures that the search index remains focused and that the retrieved context is diverse and informative.

Answer

Use a Python 'set' in a local loop on the driver node.

Answer

Manually compare every document against every other document using nested loops.

Answer

Ignore duplicates, as vector databases automatically handle them during insertion.

Question 7

Which of the following describes the 'Gold' layer in a Medallion architecture, and why is it important for GenAI data preparation?

Accepted Answer

It is the final, curated state of data, optimized for consumption by ML models.. The Gold layer represents highly refined, business-level aggregates or prepared datasets ready for consumption. In GenAI, this is where the final, cleaned, and curated training sets (or vector-ready documents) reside. Having a Gold layer ensures that models are trained on validated, high-quality data, which is fundamental to building reliable, production-grade Generative AI applications that meet organizational standards for accuracy and data governance.

Answer

It contains raw, unprocessed data for initial exploration.

Answer

It stores transient data used only for debugging intermediate steps.

Answer

It is a temporary cache for speeding up cluster startup times.

Question 8

You need to store embeddings generated by an LLM in a Delta table. Which data type is most efficient for storing these high-dimensional vector arrays in Databricks?

Accepted Answer

Store as an array of doubles.. Storing embeddings as an 'Array' of 'Floats' (or 'Doubles') in a Delta table is the most efficient and native way to handle them. This allows Spark to utilize vectorized operations for similarity search calculations and enables compatibility with various indexing libraries. Using this approach ensures high performance during retrieval, minimizing the latency of finding the most relevant context for the LLM during generation.

Answer

Store as a Base64 encoded string.

Answer

Store as separate columns for each dimension (e.g., dim1, dim2, ...).

Answer

Store as a compressed binary blob (e.g., serialized byte array).

Question 9

When preparing data for a fine-tuning task, you realize the dataset is severely imbalanced. Which Databricks technique should you use to create a more balanced dataset?

Accepted Answer

Use the sampleBy() method to perform stratified sampling.. Using Spark's 'sample()' or 'sampleBy()' transformation allows you to perform stratified oversampling or undersampling to balance classes. This ensures that the fine-tuned model doesn't become biased toward the most frequent categories in the dataset. Proper balancing is essential for ensuring robust model performance across all target classes, preventing the model from underperforming on rare but critical edge cases in real-world applications.

Answer

Run a standard 'SELECT * FROM table' query.

Answer

Increase the number of epochs during the training process.

Answer

Use a UDF to delete all rows that belong to majority classes.

Question 10

A GenAI engineer is preparing a large text corpus stored in a Unity Catalog volume for fine-tuning a chat model. The raw files are JSON Lines, each containing a 'conversation' array with alternating 'user' and 'assistant' turns, but many records have malformed turns or missing roles. The engineer needs to convert this into a Delta table with a schema of (conversation_id STRING, messages ARRAY>) while filtering out records where any turn has a null role or empty content. Which approach uses the appropriate Databricks-native capability for this transformation?

Accepted Answer

Use the from_json function with a defined StructType schema to parse each JSON line into the conversation structure, then apply a filter using a higher-order function such as filter on the messages array to retain only records where all turns have non-null role and non-empty content.. The correct approach uses from_json with an explicit schema to parse the nested JSON into the required array of structs, then applies a higher-order function to filter out records with malformed turns. This leverages Spark SQL's native, optimized JSON parsing and array operations, which scale well and enforce the target schema. It avoids the performance pitfalls of UDFs, the fragility of regex, and the indirectness of ingestion frameworks not designed for immediate nested transformation.

Answer

Load the files as a text RDD, apply a regular expression to extract role and content pairs, and then convert the RDD to a DataFrame with the desired schema using createDataFrame.

Answer

Use Databricks Auto Loader with schema evolution to ingest the files into a Bronze table, then use Delta Live Tables expectations to drop rows where the role column is null or the content column is empty.

Answer

Use the spark.read.json method with inferSchema enabled to automatically detect the schema, then use a UDF written in Python to iterate over each turn and remove records with null roles or empty content.

Question 11

You are preparing a large corpus of customer support emails stored as Parquet files in a Unity Catalog volume. The emails must be cleaned by removing boilerplate signatures and disclaimers before tokenization for a fine-tuning dataset. Your team wants to enforce this transformation as a declarative, testable pipeline stage that fails fast if any cleaned record still contains a known boilerplate marker. Which Databricks capability should you use to implement this cleaning step with built-in data quality expectations?

Accepted Answer

Define the cleaning logic in a Delta Live Tables pipeline and attach an EXPECT constraint that asserts no boilerplate marker remains in the cleaned column.. Delta Live Tables expectations are the declarative, testable mechanism for enforcing data quality during preparation. Attaching an EXPECT constraint to the cleaned table ensures the pipeline fails fast if any boilerplate marker remains, while capturing metrics for auditing. This approach provides lineage, observability, and reproducibility, unlike imperative notebooks or asynchronous alerts.

Answer

Use Spark Structured Streaming with a foreachBatch function that calls a Python UDF to strip signatures and raises an exception if a marker remains.

Answer

Create a Databricks SQL query that applies regexp_replace to the email body and schedule it as a SQL warehouse alert when a marker is detected.

Answer

Write a notebook that reads the Parquet files, applies a pandas UDF to clean the text, and writes the result back to the volume, then manually review the output.

Question 12

A data engineer is preparing a dataset of product descriptions for embedding generation. The source table in Unity Catalog contains a column 'description' with mixed languages, and the team wants to filter to English-only text before vectorization. They need a scalable, built-in Databricks function that can detect the language of each description without external API calls. Which function should they use?

Accepted Answer

ai_detect_language. The built-in ai_detect_language function is purpose-built for language identification at scale within Databricks. It returns a language code that can be used to filter the dataset to English-only records before embedding generation. Other AI functions like ai_analyze_sentiment or ai_classify serve different purposes, and regex cannot reliably determine language.

Answer

regexp_extract

Answer

ai_classify

Answer

ai_analyze_sentiment

Question 13

A GenAI engineer is preparing a corpus of HTML product pages stored in a Unity Catalog volume for a RAG application. The pages contain navigation bars, script tags, and boilerplate footers that add noise to embeddings. Which Databricks-native approach best removes this noise while preserving the main article text before chunking?

Accepted Answer

Use an ai_parse_document or BeautifulSoup-style parser in a PySpark UDF to extract the main content, then write the cleaned text as a Delta table with a content column.. Cleaning HTML before chunking requires a parser that understands document structure, not a regex or a numeric feature transformer. Extracting the main content with a document-aware parser and persisting the cleaned text to Delta produces a reliable, lineage-tracked input for chunking and embedding, which is the goal of data preparation for RAG on Databricks.

Answer

Register the volume as a Delta table and run a SQL query using regexp_replace to strip all tags, then tokenize the result with the built-in Databricks tokenizer.

Answer

Increase the chunk size to 4096 tokens so that boilerplate is a smaller proportion of each chunk, and rely on the embedding model to ignore the noise.

Answer

Load the HTML files with spark.read.text and apply a VectorAssembler to convert the raw strings into dense vectors, then cluster and drop outliers.

Question 14

You are preparing a large text corpus for fine-tuning a generative AI model. The corpus is stored in a Delta table with columns: doc_id, raw_text, and metadata. You need to create a cleaned dataset that removes personally identifiable information (PII) and normalizes whitespace, while preserving document boundaries for training. Which two actions should you perform to achieve this in a scalable and maintainable way? (Choose two.)

Accepted Answer

Apply a regular expression to replace all whitespace sequences with a single space in the raw_text column.. ai_mask redacts PII in place, and regexp_replace normalizes whitespace, together achieving the cleaning goals at scale. Both are declarative, repeatable, and preserve document boundaries. Sentiment filtering, manual inspection, and classification are either irrelevant or destructive to the dataset, and they do not address PII removal and whitespace normalization.

Answer

Convert the Delta table to Parquet files and manually inspect each file for PII.

Answer

Use the ai_classify function to label each document as 'clean' or 'dirty' and drop dirty ones.

Answer

Use the ai_analyze_sentiment function to filter out negative documents before training.

Question 15

A GenAI engineer must build a fine-tuning dataset from 40 TB of raw JSONL conversation logs stored in cloud object storage. The logs are immutable and only ever read once, in full, during preprocessing. Which storage and access configuration should be used to minimize cost while keeping the data readable by Spark on Databricks?

Accepted Answer

Register the cloud path as an external location in Unity Catalog and read the JSONL files directly with Spark.. Because the logs are immutable and read once in full, the lowest-cost approach is to leave them where they are and read them in place under Unity Catalog governance. Copying or converting the data creates a redundant full-size copy and extra compute for no reuse benefit, while legacy mounts sacrifice governance without reducing storage or access cost.

Answer

Convert the JSONL to Parquet with an external tool, upload the Parquet back, then read it from the bucket.

Answer

Run COPY INTO to ingest the JSONL into a managed Delta table, then read the Delta table for preprocessing.

Answer

Mount the object storage bucket to DBFS with a legacy mount and read files through the /mnt path.