CCNA Data Preparation Questions

PII must be removed from the content itself before training. Distributed detectors such as Presidio or Spark NLP in pandas UDFs, and model-based rewriting with ai_query(), both transform the text at scale inside Databricks. Column masks, encryption, and retention policies govern access or lifecycle but leave the underlying tokens intact, so they do not satisfy the preprocessing requirement.

The variant data type stores JSON without a rigid schema, and variant_get extracts fields dynamically, accommodating schema evolution. This avoids brittle predefined schemas and manual updates. Other options either require static schemas, rely on sampling, or only handle arrays, making them less robust for evolving nested JSON.

To update embeddings only for changed tickets, the pipeline must consume a change feed and apply changes to the embeddings table. APPLY CHANGES merges inserts, updates, and deletes while allowing embedding derivation to run only on changed rows. Full refreshes recompute everything, and file-level or query-level filters cannot persist decisions across updates.

The correct approach uses from_json with an explicit schema to parse the nested JSON into the required array of structs, then applies a higher-order function to filter out records with malformed turns. This leverages Spark SQL's native, optimized JSON parsing and array operations, which scale well and enforce the target schema. It avoids the performance pitfalls of UDFs, the fragility of regex, and the indirectness of ingestion frameworks not designed for immediate nested transformation.

Preserving document structure requires a parser that extracts layout elements rather than flattening everything to plain text, and ai_parse_document provides that for PDFs and images. Provenance requires carrying the source file path on each chunk so retrieval results can cite the originating document. Together these choices keep structure intact and make every chunk traceable.

ai_mask redacts PII in place, and regexp_replace normalizes whitespace, together achieving the cleaning goals at scale. Both are declarative, repeatable, and preserve document boundaries. Sentiment filtering, manual inspection, and classification are either irrelevant or destructive to the dataset, and they do not address PII removal and whitespace normalization.

Delta Live Tables expectations are the declarative, testable mechanism for enforcing data quality during preparation. Attaching an EXPECT constraint to the cleaned table ensures the pipeline fails fast if any boilerplate marker remains, while capturing metrics for auditing. This approach provides lineage, observability, and reproducibility, unlike imperative notebooks or asynchronous alerts.

Near-duplicate detection at scale needs a compact per-record signature plus a candidate-generation strategy, which MinHash with locality-sensitive hashing banding provides. Exact hashing misses near-duplicates, embedding clustering is costly and nondeterministic, and length-based filtering has no content signal, so none of those satisfy the deterministic, scalable requirement.

The built-in ai_detect_language function is purpose-built for language identification at scale within Databricks. It returns a language code that can be used to filter the dataset to English-only records before embedding generation. Other AI functions like ai_analyze_sentiment or ai_classify serve different purposes, and regex cannot reliably determine language.

Unity Catalog provides centralized governance, including automated data lineage tracking. This allows engineers to trace the source of data used for model training, which is vital for reproducibility and regulatory compliance. Knowing the exact provenance of training data helps in troubleshooting model bias and ensures that sensitive data sources are correctly identified and managed throughout the lifecycle of the model training process.

Storing embeddings as an 'Array' of 'Floats' (or 'Doubles') in a Delta table is the most efficient and native way to handle them. This allows Spark to utilize vectorized operations for similarity search calculations and enables compatibility with various indexing libraries. Using this approach ensures high performance during retrieval, minimizing the latency of finding the most relevant context for the LLM during generation.

Cleaning HTML before chunking requires a parser that understands document structure, not a regex or a numeric feature transformer. Extracting the main content with a document-aware parser and persisting the cleaned text to Delta produces a reliable, lineage-tracked input for chunking and embedding, which is the goal of data preparation for RAG on Databricks.

Delta Sync indexes in Databricks Vector Search rely on Change Data Feed to pick up inserts, updates, and deletes incrementally. Choosing a triggered or continuous sync mode keeps embeddings current without full recomputation. Enabling CDF on the source Delta table is a prerequisite, and this pattern is the documented way to keep a RAG index fresh as documents change.

When the embedding model is English-only, the pipeline must identify which rows need translation. Detecting language per row and storing the result as a column gives a deterministic routing signal, so English rows embed directly while non-English rows are sent to translation. This keeps the preparation step auditable and avoids both data loss and unnecessary translation cost.

The Gold layer represents highly refined, business-level aggregates or prepared datasets ready for consumption. In GenAI, this is where the final, cleaned, and curated training sets (or vector-ready documents) reside. Having a Gold layer ensures that models are trained on validated, high-quality data, which is fundamental to building reliable, production-grade Generative AI applications that meet organizational standards for accuracy and data governance.

Because the logs are immutable and read once in full, the lowest-cost approach is to leave them where they are and read them in place under Unity Catalog governance. Copying or converting the data creates a redundant full-size copy and extra compute for no reuse benefit, while legacy mounts sacrifice governance without reducing storage or access cost.

To reduce redundancy and ensure language consistency, the engineer should use MinHash with LSH to remove near-duplicate documents and apply language detection to filter to the target language. These steps directly address the stated goals and are scalable for large corpora.

Delta Lake provides ACID transactions and time travel, which are essential for robust data engineering. ACID transactions ensure that data preparation pipelines never leave tables in a partially written state, while time travel allows for auditing and reverting changes. These features significantly simplify data lifecycle management, reduce the need for custom retry logic, and enhance the overall reliability of data preparation workflows in complex, multi-user production environments.

Using Spark's 'dropDuplicates()' on the text content column or calculating a hash (e.g., MD5) of the text to identify duplicates is highly scalable. This is crucial because redundant information in the vector database can cause the retriever to prioritize duplicate entries, leading to biased results and inefficiency. Removing duplicates ensures that the search index remains focused and that the retrieved context is diverse and informative.

The correct approach uses Spark NLP's DocumentAssembler and Normalizer within a Spark ML Pipeline, which provides distributed, reproducible text normalization. Saving the pipeline to MLflow ensures version control and reusability. This method is scalable and integrates well with Databricks, making it suitable for preparing large fine-tuning datasets with complex cleaning needs.

The most appropriate approach is to use concat_ws to join messages after prefixing each with a role token. This creates a single formatted string per conversation efficiently and deterministically, which is ideal for fine-tuning chat models.

Delta Lake on Unity Catalog is the only option that simultaneously delivers ACID transactions, time travel, schema enforcement, and column-level lineage for the prepared dataset. Parquet, CSV, and ORC files lack transactional table semantics, and the Hive metastore does not provide the Unity Catalog lineage the team needs.

Batch preprocessing of an existing Delta table is best done with Spark DataFrame transformations, which distribute token counting and filtering across executors. Writing the cleaned result to a Unity Catalog table creates a governed, reproducible training artifact. Driver-side pandas and row-by-row writes do not scale, views recompute on every read, and streaming pipelines are overkill for a one-time preparation step.

The reviews need markup removed and length controlled before embedding. Stripping HTML and normalizing whitespace cleans the text, while chunking with overlap sized to the model token limit prevents truncation of long paragraphs. Uppercasing, dropping short reviews, and archiving raw HTML do not resolve the stated noise and overflow problems.

Using Spark's 'sample()' or 'sampleBy()' transformation allows you to perform stratified oversampling or undersampling to balance classes. This ensures that the fine-tuned model doesn't become biased toward the most frequent categories in the dataset. Proper balancing is essential for ensuring robust model performance across all target classes, preventing the model from underperforming on rare but critical edge cases in real-world applications.

An append-only bronze ingestion table should be defined as a streaming table so each pipeline update consumes only new source records. This makes expensive operations like ai_query run once per new event rather than on the full history. Materialized views and batch-style definitions re-read all source data, and storage properties like Auto Optimize only affect file layout, not incremental read behavior.

File fragmentation from frequent appends is addressed by consolidating small files. OPTIMIZE with optional ZORDER compacts and clusters existing data, while auto compaction and optimized writes prevent new fragmentation from accumulating. Together they reduce the number of files the embedding job must open and can improve predicate pushdown, directly improving read throughput.

Unity Catalog's dynamic views and column-level masking policies are the standard for securing PII. By applying functions like MASK or current_user() within a view definition, administrators decouple security logic from physical table storage. This approach is essential for compliance, ensuring that sensitive data is hidden at query time without duplicating datasets, thus maintaining a single source of truth while enforcing granular access control across all Databricks workspaces and compute resources.

Selecting a chunking strategy requires balancing semantic context and technical limitations like context window size. If chunks are too small, they lack meaning; if too large, they exceed the limits of the embedding model and introduce irrelevant noise. Proper chunking is vital for maximizing the accuracy of RAG systems, as it determines the granularity of the information available for the retriever to present to the LLM.

The binaryFile data source is designed to read arbitrary files as rows with path and content, enabling distributed extraction with a UDF. This yields one row per document with text and source path in a Delta table. Driver loops and plain text readers cannot handle mixed binary formats at scale or in parallel.

Change Data Feed captures row-level changes with commit versions, allowing a downstream job to read only the inserts, updates, and deletes that occurred since its last checkpoint. This directly solves the problem of reprocessing unchanged rows. Partitioning, compaction, and timestamp filtering improve read characteristics but do not expose which rows changed, so they cannot deliver incremental processing.

Filtering rows based on whether any nested element meets a predicate is exactly what the exists higher-order function provides, and its lambda can test both role and content length in one pass. Exploding reshapes and shuffles data, string matching is ambiguous, and transform preserves array length so it cannot signal absence of a match.

Using exploratory data analysis (EDA) with Databricks SQL or Spark to analyze distribution statistics is the most effective way to identify bias. By comparing the distribution of the training set to a representative sample of real-world production data, engineers can identify under-represented categories. This ensures that the fine-tuned model performs reliably across all expected input scenarios, preventing the model from developing blind spots.

Removing boilerplate and normalizing whitespace cuts wasted tokens while sharpening the text signal, and deduplicating plus dropping undersized fragments avoids paying to embed redundant or low-information content. Uppercasing, stop-word stripping, and blind truncation either damage semantics or discard content, so they do not improve cost or retrieval quality.

Embedding models encode the exact tokens they receive, so HTML markup, irregular whitespace, and inconsistent casing add noise that dilutes semantic signal. Normalizing the text before embedding removes that noise and produces vectors that better reflect meaning. Storage format and vector dimension do not change the input text, so they cannot solve the quality problem.

The training job requires one row per conversation with the message sequence preserved, so the individual message rows must be grouped back together. collect_list aggregates messages into an ordered array per conversation, and serializing that array yields the nested format the job expects. Exploding, pivoting, or flattening would alter the structure and lose the required per-conversation layout.

Distributed tokenization of a large corpus requires the work to run inside Spark executors, which a pandas UDF provides by operating vectorized on each partition. Reading with pandas on the driver or looping over files locally centralizes the workload and breaks at scale. A pandas UDF keeps the corpus distributed while applying the tokenizer and truncation per row.

The Feature Store acts as a centralized repository for standardized, reusable features. It ensures that the same logic is used for data preparation during both training and inference. This consistency is critical for preventing training-serving skew, where the model's performance in production differs from its training performance because the input data was transformed differently, ensuring the model remains accurate and reliable in real-world scenarios.