Databricks-GenAI-Assoc Data Preparation Practice Test 2 — 15 Questions

Question 1

An enterprise engineering team is preparing unstructured PDF invoices stored in a Unity Catalog volume for a Retrieval-Augmented Generation pipeline. Which ingestion approach best aligns with Databricks-recommended data preparation practices for generative AI?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Ingest the files into a Delta table from the Unity Catalog volume, then use LangChain integration alongside Databricks Vector Search to parse and index the documents.. Utilizing standard PySpark or Auto Loader to load files from the volume into a Delta table, followed by applying vector search indexing, ensures secure governance and scalable ingestion. Unity Catalog provides robust access control for the underlying unstructured files. Managing unstructured data through Delta tables and Unity Catalog maintains enterprise data lineage and simplifies downstream document parsing pipelines.

Answer

Manually copy all PDF files to the local driver node of a standard interactive cluster and execute custom Python scripts to parse them into local CSVs.

Answer

Upload the invoices directly into a legacy Hive metastore table as binary blobs without leveraging Unity Catalog security features.

Answer

Store the PDF binaries inside an unmanaged cloud bucket, bypassing Databricks entirely until the model inference stage requires document retrieval.

Question 2

A data engineer is preparing a large set of policy PDFs stored in a Unity Catalog volume for a RAG application. Before generating embeddings, they must remove page headers, footers, and page numbers that repeat across nearly every page. Which approach is most appropriate for cleaning this text at scale on Databricks?

Accepted Answer

Use the ai_parse_document function to extract text and layout from each PDF, then apply rule-based filtering to remove recurring header and footer patterns.. The most efficient and accurate method is to use ai_parse_document to extract text with layout information, then apply rule-based filtering to remove repeating headers and footers. This leverages Databricks-native capabilities and scales across many documents, ensuring clean text for embedding generation.

Answer

Use the binaryFile data source to read raw bytes and apply a regular expression to strip everything before the first newline and after the last newline.

Answer

Convert each PDF page to an image, run OCR with a third-party library, and manually delete header and footer regions using bounding box coordinates.

Answer

Load the PDFs with spark.read.format("pdf") and drop the first and last line of each page based on line index.

Question 3

A GenAI team is preparing a fine-tuning dataset stored as a Delta table in Unity Catalog. They need to ensure that each training example is a valid JSON string conforming to a specific schema before feeding it to the model. They want to validate and optionally quarantine malformed records in a single pass. Which Databricks SQL function is most appropriate for this task?

Accepted Answer

from_json. The from_json function parses a JSON string into a struct using a specified schema, and returns null if the string does not conform. By applying from_json and filtering on null, the team can validate and quarantine malformed records in one pass, ensuring only valid examples are used for fine-tuning.

Answer

schema_of_json

Answer

json_tuple

Answer

to_json

Question 4

A data engineer is preparing a large corpus of text for a RAG application. The text is stored in a Delta table with columns doc_id and content. The engineer needs to split the content into chunks of approximately 512 tokens with a 50-token overlap, and then compute embeddings for each chunk. They want to use Databricks-native capabilities to minimize data movement and cost. Which two approaches are valid and efficient for this task? (Choose two.)

Accepted Answer

Use Spark NLP's DocumentAssembler and Tokenizer to split text into chunks, then use a pre-trained SentenceTransformer within a Pandas UDF to compute embeddings.. Using Databricks Foundation Model APIs or Spark NLP with Pandas UDFs allows distributed, in-cluster processing of both chunking and embedding. These methods avoid data movement and leverage Spark's scalability, making them efficient for large RAG corpora.

Answer

Use the Databricks Vector Search API to directly index the raw text without chunking, relying on its built-in chunking feature.

Answer

Export the text to a Parquet file on cloud storage, then use a separate Python script with the Hugging Face transformers library on a single node to compute embeddings.

Answer

Use a Scala UDF that calls an external embedding API for each row, then write the results to a Delta table.

Question 5

A data engineer is preparing a dataset for training a generative AI model. The dataset contains a column 'text' with raw user reviews. Before tokenization, the engineer needs to remove HTML tags and special characters, convert text to lowercase, and expand common contractions. Which Databricks feature or approach is best suited for this text normalization step?

Accepted Answer

Use Spark NLP's DocumentNormalizer and Normalizer annotators in a Spark ML pipeline.. Spark NLP's DocumentNormalizer and Normalizer are designed for exactly these text cleaning tasks, providing scalable, distributed processing within Databricks. They handle HTML tag removal, lowercasing, and special character removal, and can be customized for contraction expansion.

Answer

Use the ai_query function with a prompt to ask an LLM to normalize the text.

Answer

Use Delta Live Tables expectations to enforce that text is normalized.

Answer

Use the Databricks Feature Store to apply normalization functions during feature computation.

Question 6

A data engineer is preparing a dataset for a RAG application. The source data is a Delta table with a 'content' column containing long documents. To improve retrieval, they need to split each document into overlapping chunks of approximately 500 tokens. They want to use a built-in Databricks function that leverages a tokenizer to count tokens accurately. Which approach should they take?

Accepted Answer

Use the ai_chunk function with a specified chunk size and overlap.. ai_chunk is the built-in Databricks function that performs token-aware chunking with overlap, directly satisfying the RAG preparation requirement. Other methods either rely on words or characters, which do not align with token limits, or require custom code that is not built-in and may be less efficient.

Answer

Use a Python UDF that calls the tiktoken library to count tokens and manually slice the string.

Answer

Use the substring function to extract the first 500 characters of each document.

Answer

Use the split function to divide the content by spaces and then group every 500 words.

Question 7

A team prepares customer support transcripts for fine-tuning. Each transcript is a JSON record with a nested messages array, and approximately 8 percent of records contain a null role field. The team wants records with malformed messages to be quarantined rather than silently dropped. Which Delta Live Tables configuration accomplishes this?

Accepted Answer

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL), and create a separate streaming table that selects records where role IS NULL from the source.. Quarantining malformed records requires keeping them somewhere queryable. An EXPECT constraint without an ON VIOLATION clause preserves violating rows in the primary table and logs the violation, while a second streaming table can select the malformed records into a dedicated quarantine dataset. Drop and fail behaviors both remove the ability to review the bad data.

Answer

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL) ON VIOLATION DROP ROW, and inspect the event log for dropped records.

Answer

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL) ON VIOLATION DROP ROW, and configure the pipeline to write dropped rows to a Unity Catalog volume automatically.

Answer

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL) ON VIOLATION FAIL UPDATE, and rely on pipeline retries to reprocess the batch.

Question 8

A data engineer is building a RAG pipeline and needs to store document embeddings in Databricks Vector Search. The embeddings are computed from text chunks and stored in a Delta table with columns chunk_id, embedding (array of floats), and metadata. The engineer wants to enable fast similarity search with metadata filtering. Which step is required to make this Delta table searchable in Databricks Vector Search?

Accepted Answer

Enable Change Data Feed on the Delta table and then create a Vector Search index with the Delta table as the source.. Databricks Vector Search indexes are built from Delta tables with Change Data Feed enabled to capture row-level changes and keep the index synchronized. This allows the index to reflect updates, inserts, and deletes in near real-time, enabling fast similarity search with metadata filtering.

Answer

Convert the Delta table to a Parquet file and then create a Vector Search index from the Parquet file.

Answer

Create a Vector Search index directly from the Delta table without any additional configuration.

Answer

Use the Databricks Feature Store to publish the embeddings as a feature table and then create a Vector Search index from the feature table.

Question 9

A generative AI team stores raw support tickets in a Unity Catalog volume as newline-delimited JSON. Before fine-tuning, they must convert the nested arrays into a flat table and enforce the ticket schema on every read so malformed records fail fast instead of silently producing nulls. Which approach best satisfies both requirements?

Accepted Answer

Read the files with spark.read.json using an explicit StructType, then call explode on the nested columns.. Enforcing a schema at read time requires an explicit StructType, which makes malformed records fail instead of being coerced to nulls. Converting nested arrays into rows requires explode, which emits one row per array element. Combining an explicit schema with explode satisfies both the strict enforcement and the flattening requirement in a single pass.

Answer

Read the files with spark.read.json using inferSchema, then call flatten on the nested columns.

Answer

Read the files with spark.read.json using an explicit StructType, then call flatten on the nested columns.

Answer

Read the files with spark.read.text, then apply from_json with the schema and use explode_outer on the parsed columns.

Question 10

A GenAI engineer has a Delta table of 50 million customer support transcripts in Unity Catalog, each containing a raw_text column. Before generating embeddings with a foundation model endpoint, they must split each transcript into ~500-token chunks and preserve the source document_id and chunk index. Which Databricks-native approach should they use to produce a new Delta table of chunked records at scale?

Accepted Answer

Define a pandas UDF or mapInPandas that applies the chunking function per partition, add a monotonically_increasing_id for chunk index, and write the result as a Delta table.. Chunking must be distributed because the dataset is large and the operation is per-row. A pandas UDF or mapInPandas keeps the transformation inside Spark, parallelizes the chunker, and writes partitioned results to Delta. This preserves document_id and chunk index for later retrieval, and avoids driver memory pressure or external data movement.

Answer

Export the table to Parquet, run a separate Python script on a single VM with LangChain's RecursiveCharacterTextSplitter, and re-import the chunks into Delta.

Answer

Use spark.read.table("transcripts").repartition(1).collect() and chunk the text in a Python UDF on the driver, then write the results back to Delta.

Answer

Use the ai_prep_chunks() SQL function in a CREATE TABLE AS SELECT statement to automatically chunk the raw_text column.

Question 11

An engineer is building a training set from a Delta table in Unity Catalog. The table contains a free-text column with customer emails, and the fine-tuning data must not expose personal data. Which approach satisfies the requirement while keeping the data usable for model training?

Accepted Answer

Apply a Unity Catalog column mask on the email-bearing column that replaces detected personal identifiers with synthetic placeholders for non-privileged principals.. The goal is to keep the dataset trainable while preventing personal data from reaching the model. A Unity Catalog column mask transforms sensitive values based on the caller's identity, so the fine-tuning job sees redacted text and authorized users retain full access. Access revocation, column dropping, and time travel do not redact the content itself.

Answer

Enable Delta Lake time travel and instruct the training job to read a specific table version that predates the ingestion of personal data.

Answer

Copy the table to a new schema, drop the email column, and train on the remaining columns.

Answer

Grant SELECT on the table only to the service principal that runs the fine-tuning job, and revoke access for all human users.

Question 12

A data engineer needs to convert a Delta table of raw HTML product descriptions into clean plain text for embedding generation. The cleaning must remove HTML tags, collapse whitespace, and normalize Unicode. Which Databricks capability should they use to apply this transformation across millions of rows?

Accepted Answer

Write a Spark SQL query using regexp_replace and trim functions directly on the raw column and save the result to a new Delta table.. Distributed string transformations are best expressed with Spark SQL functions because they run natively on executors and avoid Python overhead. regexp_replace, trim, and translate handle tag removal, whitespace collapse, and Unicode normalization. Writing the cleaned output to a new Delta table keeps the raw HTML available for reprocessing if cleaning rules change.

Answer

Create a materialized view with the html_clean() function and let Databricks automatically rewrite the stored data.

Answer

Mount the Delta table as a DBFS file and run a shell script with sed and awk on each node to clean the text.

Answer

Use the Databricks Assistant to rewrite the HTML content in the notebook and copy the cleaned text into the table.

Question 13

A Data Engineer needs to ensure that PII data in a Delta table is masked before serving it to non-privileged users. Which Databricks feature provides the most efficient, centralized control for this requirement?

Accepted Answer

Utilize Unity Catalog column-level masking functions.. Unity Catalog's dynamic views and column-level masking policies are the standard for securing PII. By applying functions like MASK or current_user() within a view definition, administrators decouple security logic from physical table storage. This approach is essential for compliance, ensuring that sensitive data is hidden at query time without duplicating datasets, thus maintaining a single source of truth while enforcing granular access control across all Databricks workspaces and compute resources.

Answer

Apply Row-Level Security filters using Delta table constraints.

Answer

Create materialized views with hard-coded redacted values.

Answer

Implement Spark UDFs to perform in-memory data masking.

Question 14

You need to ingest data from an external JSON source into a Delta table. The source schema is inconsistent. Which strategy is most effective for preparing this data?

Accepted Answer

Implement a bronze-silver medallion pattern.. The 'bronze-to-silver' pattern is the industry standard in Databricks for handling inconsistent data. By loading raw JSON into a 'Bronze' table with a schema of type 'string' (or a single JSON column), you preserve all information for audit. You then use 'Silver' tables to perform schema enforcement, data cleaning, and type casting. This separation of concerns allows for robust error handling and iterative refinement of the data cleaning logic.

Answer

Use inferSchema = true in the read configuration.

Answer

Define a rigid schema at the ingestion point.

Answer

Convert the JSON to CSV before loading into Delta.

Question 15

Which TWO of the following are benefits of using Delta Lake for data preparation over standard Parquet files on cloud storage?

Accepted Answer

ACID compliance for reliable write operations.. Delta Lake provides ACID transactions and time travel, which are essential for robust data engineering. ACID transactions ensure that data preparation pipelines never leave tables in a partially written state, while time travel allows for auditing and reverting changes. These features significantly simplify data lifecycle management, reduce the need for custom retry logic, and enhance the overall reliability of data preparation workflows in complex, multi-user production environments.

Answer

Automatic conversion of JSON to XML format.

Answer

Native support for cross-cloud streaming ingestion.

Answer

Automatic hardware upgrades for compute clusters.