Databricks-GenAI-Assoc Data Preparation Practice Test 4 — 15 Questions

Question 1

A Generative AI engineer is building a Delta Live Tables pipeline that ingests raw JSON event logs into a bronze table, then uses ai_query to classify each event's free-text field. The classification call is expensive, so the engineer wants to avoid re-running it on events that have already been processed in previous pipeline updates. The source table is append-only and new events arrive continuously. Which Delta Live Tables feature should the engineer configure on the bronze table to prevent reprocessing of previously ingested rows?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure the bronze table as a streaming table so it processes only new data on each pipeline update.. An append-only bronze ingestion table should be defined as a streaming table so each pipeline update consumes only new source records. This makes expensive operations like ai_query run once per new event rather than on the full history. Materialized views and batch-style definitions re-read all source data, and storage properties like Auto Optimize only affect file layout, not incremental read behavior.

Answer

Add a QUALIFY clause that filters out rows where the classification column is already populated.

Answer

Enable change data capture by setting pipelines.cdcEnabled to true on the bronze table.

Answer

Set the table property pipelines.autoOptimize.managed to true on the bronze table.

Question 2

A team is preparing a pretraining corpus and must filter out near-duplicate documents before tokenization. The corpus contains 500 million short text records in a Delta table. They want a scalable, deterministic deduplication signal that can be computed per record and compared across the dataset. Which approach best fits this requirement?

Accepted Answer

Compute a MinHash signature per document with a fixed number of hash permutations and compare signatures with a locality-sensitive hashing banding scheme.. Near-duplicate detection at scale needs a compact per-record signature plus a candidate-generation strategy, which MinHash with locality-sensitive hashing banding provides. Exact hashing misses near-duplicates, embedding clustering is costly and nondeterministic, and length-based filtering has no content signal, so none of those satisfy the deterministic, scalable requirement.

Answer

Sort the corpus by document length and drop every record whose length falls within one standard deviation of another record.

Answer

Compute an MD5 hash of each full document and drop rows sharing the same hash value.

Answer

Train a sentence embedding model and cluster documents with k-means, then drop all but one document per cluster.

Question 3

A data engineer is preparing a large corpus of customer support transcripts stored as Parquet in a Unity Catalog volume. Before generating embeddings, each transcript must be tokenized and truncated to a maximum token length. The engineer wants to use a Databricks-native approach that runs distributed across the cluster and avoids pulling the full corpus into a single node. Which approach best satisfies these requirements?

Accepted Answer

Use spark.read.parquet to load the corpus and apply a pandas_udf that tokenizes and truncates each transcript.. Distributed tokenization of a large corpus requires the work to run inside Spark executors, which a pandas UDF provides by operating vectorized on each partition. Reading with pandas on the driver or looping over files locally centralizes the workload and breaks at scale. A pandas UDF keeps the corpus distributed while applying the tokenizer and truncation per row.

Answer

Use dbutils.fs.cp to copy the Parquet files to a local path, then run a Python tokenizer loop over each file.

Answer

Create a Delta table from the Parquet files and query it with a SQL UDF that calls a tokenizer from a Python library.

Answer

Read the Parquet files with pandas.read_parquet on the driver, tokenize, then write the truncated text back to the volume.

Question 4

A Generative AI engineer is preparing a Delta table of product reviews for a retrieval-augmented generation application. The reviews contain HTML tags, inconsistent casing, and occasional very long paragraphs that exceed the embedding model's context window. The engineer wants to clean and normalize the text before chunking and embedding. Which two actions should the engineer take to directly address the stated quality issues? (Choose two.)

Accepted Answer

Split long paragraphs into overlapping chunks sized to the embedding model's token limit.. The reviews need markup removed and length controlled before embedding. Stripping HTML and normalizing whitespace cleans the text, while chunking with overlap sized to the model token limit prevents truncation of long paragraphs. Uppercasing, dropping short reviews, and archiving raw HTML do not resolve the stated noise and overflow problems.

Answer

Drop every review shorter than 100 characters to reduce noise in the corpus.

Answer

Store the raw HTML in a separate column so it remains available for future parsing needs.

Answer

Convert all review text to uppercase to standardize casing across the corpus.

Question 5

A data engineer needs to persist a prepared instruction-tuning dataset so that downstream fine-tuning jobs can read it with ACID guarantees, time travel, and schema enforcement, and so that Unity Catalog can track column-level lineage. Which storage format and registration should be used?

Accepted Answer

Write the dataset as a Delta table registered in Unity Catalog.. Delta Lake on Unity Catalog is the only option that simultaneously delivers ACID transactions, time travel, schema enforcement, and column-level lineage for the prepared dataset. Parquet, CSV, and ORC files lack transactional table semantics, and the Hive metastore does not provide the Unity Catalog lineage the team needs.

Answer

Write the dataset as an ORC table in the Hive metastore.

Answer

Write the dataset as CSV files to an external location registered in Unity Catalog.

Answer

Write the dataset as Parquet files in a Unity Catalog volume.

Question 6

A team stores raw documents in a Unity Catalog volume and needs to build a training set for fine-tuning a large language model. The raw files are in mixed formats including PDF, DOCX, and plain text. The team wants a single Delta table where every row is one document with its extracted text and source path, and wants the extraction to run in parallel across the cluster. Which approach should the team use?

Accepted Answer

Use binaryFile data source to read the volume, then apply a UDF that extracts text per file and writes results to a Delta table.. The binaryFile data source is designed to read arbitrary files as rows with path and content, enabling distributed extraction with a UDF. This yields one row per document with text and source path in a Delta table. Driver loops and plain text readers cannot handle mixed binary formats at scale or in parallel.

Answer

Mount the volume as a local filesystem path and use a Python for-loop to read and extract each file on the driver.

Answer

Use spark.read.text on the volume, which automatically parses PDF and DOCX content into text columns.

Answer

Create an external table over the volume with a schema that includes columns for each document format's fields.

Question 7

You are building a data preparation pipeline for a generative AI application. The raw data is stored in a Unity Catalog volume as JSON files with nested fields. You need to flatten the nested structure and extract specific fields into a Delta table for downstream embedding. The JSON schema may evolve, with new fields added occasionally. Which approach provides the most robust and maintainable solution?

Accepted Answer

Use the variant data type to store the JSON and then use variant_get to extract fields.. The variant data type stores JSON without a rigid schema, and variant_get extracts fields dynamically, accommodating schema evolution. This avoids brittle predefined schemas and manual updates. Other options either require static schemas, rely on sampling, or only handle arrays, making them less robust for evolving nested JSON.

Answer

Use the explode function on the nested arrays and then select the required fields.

Answer

Use the from_json function with a predefined schema and select the required fields.

Answer

Use the schema_of_json function to infer the schema at runtime and then apply from_json.

Question 8

An engineer is cleaning a Delta table of customer support transcripts stored in Unity Catalog. The table has a nested array column named messages, where each element has role and content fields. They need to keep only rows where at least one message has role equal to 'user' and content longer than 20 characters. Which transformation correctly expresses this filter?

Accepted Answer

Use the higher-order function exists on the messages array with a lambda testing role = 'user' AND length(content) > 20.. Filtering rows based on whether any nested element meets a predicate is exactly what the exists higher-order function provides, and its lambda can test both role and content length in one pass. Exploding reshapes and shuffles data, string matching is ambiguous, and transform preserves array length so it cannot signal absence of a match.

Answer

Cast the messages array to a string and apply a LIKE pattern matching the literal text 'user' followed by any content.

Answer

Use the transform higher-order function on messages and then check whether the resulting array is non-empty.

Answer

Explode the messages array, filter on role and content length, then group by the row identifier to rebuild the rows.

Question 9

A Databricks team is building a retrieval corpus from mixed-format documents stored in a Unity Catalog volume. They need a preparation pipeline that preserves document structure for later chunking and that records which source file each chunk came from so retrieval results can cite evidence. Which two design choices best meet these requirements? (Choose two.)

Accepted Answer

Carry the source file path as a metadata column on each chunk and store it in the Delta table alongside the chunk text and embedding.. Preserving document structure requires a parser that extracts layout elements rather than flattening everything to plain text, and ai_parse_document provides that for PDFs and images. Provenance requires carrying the source file path on each chunk so retrieval results can cite the originating document. Together these choices keep structure intact and make every chunk traceable.

Answer

Concatenate all documents into a single large text field before chunking to reduce the number of rows.

Answer

Convert every document to plain text and discard page and heading markers to normalize the corpus.

Answer

Generate embeddings for entire documents rather than chunks to avoid storing multiple rows per file.

Question 10

A Generative AI engineer is preparing a Delta table of support tickets for embedding generation. The pipeline computes embeddings with ai_query and stores them in a separate embeddings table keyed by ticket_id. Tickets are frequently updated by agents, and the engineer wants the embeddings table to reflect the latest ticket text without recomputing embeddings for tickets whose text has not changed. Which design best achieves this?

Accepted Answer

Use change data capture with APPLY CHANGES to merge ticket updates into the embeddings table, recomputing embeddings only for changed rows.. To update embeddings only for changed tickets, the pipeline must consume a change feed and apply changes to the embeddings table. APPLY CHANGES merges inserts, updates, and deletes while allowing embedding derivation to run only on changed rows. Full refreshes recompute everything, and file-level or query-level filters cannot persist decisions across updates.

Answer

Add a hash column of the ticket text and use a QUALIFY clause to filter unchanged rows within the same query.

Answer

Enable Auto Optimize on the embeddings table so only modified files are rewritten during updates.

Answer

Run a full refresh of the embeddings table on every pipeline update so all ticket embeddings stay current.

Question 11

A team is building a RAG application using Databricks Vector Search with a Delta table as the source. They need the index to automatically reflect new and updated chunks as the source table changes, without rebuilding the entire index each time. Which configuration should they use?

Accepted Answer

Create a Vector Search index with sync mode set to TRIGGERED or CONTINUOUS and specify a Delta table with Change Data Feed enabled as the source.. Delta Sync indexes in Databricks Vector Search rely on Change Data Feed to pick up inserts, updates, and deletes incrementally. Choosing a triggered or continuous sync mode keeps embeddings current without full recomputation. Enabling CDF on the source Delta table is a prerequisite, and this pattern is the documented way to keep a RAG index fresh as documents change.

Answer

Enable Delta Lake time travel on the source table and point the Vector Search index at a specific version using VERSION AS OF.

Answer

Use an external HNSW index built on a Databricks cluster and register it in Unity Catalog as a model, then query it with the model serving endpoint.

Answer

Create a standard Vector Search index and schedule a nightly job that drops the index and recreates it from the full Delta table.

Question 12

A team is preparing a large text corpus for embedding generation with a foundation model endpoint on Databricks. They must reduce token cost and improve retrieval quality before vectorization. Which two preprocessing steps should be applied to the raw text? (Choose two.)

Accepted Answer

Deduplicate near-identical passages and drop documents that fail a minimum content-length threshold.. Removing boilerplate and normalizing whitespace cuts wasted tokens while sharpening the text signal, and deduplicating plus dropping undersized fragments avoids paying to embed redundant or low-information content. Uppercasing, stop-word stripping, and blind truncation either damage semantics or discard content, so they do not improve cost or retrieval quality.

Answer

Remove stop words and punctuation from every passage before generating embeddings.

Answer

Truncate every document to its first 512 characters to guarantee uniform chunk size.

Answer

Convert all text to uppercase before tokenization to standardize casing.

Question 13

A data engineer needs to prepare a Delta table of customer reviews for embedding generation. The reviews contain HTML tags, inconsistent whitespace, and mixed casing that hurt embedding quality. Which preparation step should be applied before generating embeddings?

Accepted Answer

Normalize the text by stripping HTML tags, collapsing whitespace, and applying consistent casing.. Embedding models encode the exact tokens they receive, so HTML markup, irregular whitespace, and inconsistent casing add noise that dilutes semantic signal. Normalizing the text before embedding removes that noise and produces vectors that better reflect meaning. Storage format and vector dimension do not change the input text, so they cannot solve the quality problem.

Answer

Apply a hash of the raw review text as the embedding input to guarantee deterministic vectors.

Answer

Increase the embedding vector dimension to capture the additional characters in the raw text.

Answer

Store the reviews in a Parquet file instead of Delta to improve text compression before embedding.

Question 14

A data engineer is preparing a large corpus of support tickets stored in a Unity Catalog volume for fine-tuning a Llama model on Databricks. They must remove personally identifiable information (PII) before the data reaches the training cluster. Which TWO approaches are appropriate for detecting and redacting PII at scale in this pipeline? (Choose two.)

Accepted Answer

Apply a Spark NLP or presidio-based pandas UDF that detects entities such as names, emails, and phone numbers and replaces them with placeholders.. PII must be removed from the content itself before training. Distributed detectors such as Presidio or Spark NLP in pandas UDFs, and model-based rewriting with ai_query(), both transform the text at scale inside Databricks. Column masks, encryption, and retention policies govern access or lifecycle but leave the underlying tokens intact, so they do not satisfy the preprocessing requirement.

Answer

Set the table's retention policy to 0 days so that raw tickets are deleted immediately after ingestion.

Answer

Use Unity Catalog column masks on the raw text column so that anyone querying the table sees redacted values.

Answer

Enable server-side encryption on the Unity Catalog volume and rely on the storage layer to anonymize the text.

Question 15

A GenAI engineer is preparing a large text corpus for fine-tuning an LLM. The corpus contains many near-duplicate documents and documents in multiple languages. They need to reduce redundancy and ensure language consistency. Which two steps should be performed during data preparation? (Choose two.)

Accepted Answer

Use the language detection function from the spark-nlp library to filter documents to a single target language.. To reduce redundancy and ensure language consistency, the engineer should use MinHash with LSH to remove near-duplicate documents and apply language detection to filter to the target language. These steps directly address the stated goals and are scalable for large corpora.

Answer

Train a custom tokenizer on the entire corpus to better handle multilingual text.

Answer

Use Delta Lake time travel to revert to a previous version of the dataset if duplicates are found.

Answer

Apply a fixed-size chunking strategy to split all documents into 512-token segments before deduplication.