Databricks-DE-Pro Data Transformation, Cleansing, Quality Practice Question
A Data Engineer is tasked with cleaning a dataset in Databricks. The dataset contains a column 'phone_number' with various formats, including parentheses, dashes, and spaces. The engineer needs to standardize all phone numbers to a digits-only format (e.g., '1234567890'). Which approach is most efficient and scalable?
⚠ Common exam trap
The trap here is opting for a Python UDF because it seems familiar, but UDFs are slower and should be avoided when built-in functions suffice.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use the regexp_replace() function in Spark SQL to remove all non-digit characters.
regexp_replace() is a built-in, distributed function that efficiently removes all non-digit characters in one step. It leverages Spark's optimized execution engine and is the most scalable and straightforward method for this cleansing task. Other options involve less efficient UDFs, incomplete character replacement, or complex splitting logic.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use a Python UDF with the re.sub() method to strip non-digit characters.
Why it's wrong here
While a Python UDF can achieve the same result, UDFs are less efficient than built-in functions because they prevent Spark's Catalyst optimizer from optimizing the query and require serialization overhead. For a simple string replacement, using regexp_replace() is preferred for performance and scalability.
- ✗
Use the translate() function to replace parentheses and dashes with empty strings.
Why it's wrong here
translate() can replace specific characters, but it requires listing every non-digit character to remove, which is error-prone and not comprehensive. It would not handle all possible formatting variations (e.g., dots, spaces, plus signs) without an exhaustive list. regexp_replace() with a pattern is more robust.
- ✓
Use the regexp_replace() function in Spark SQL to remove all non-digit characters.
Why this is correct
regexp_replace() is a built-in Spark SQL function that can remove all non-digit characters from a string using a regular expression like '[^0-9]'. It operates in a distributed manner, making it efficient for large datasets. This is the simplest and most scalable approach for standardizing phone numbers.
- ✗
Use the split() function to split on non-digit characters and then concatenate the parts.
Why it's wrong here
split() followed by concatenation is cumbersome and may not correctly handle consecutive non-digit characters or leading/trailing delimiters. It also requires additional transformations, making it less efficient and more complex than a single regexp_replace() call. This approach is not recommended for standardizing phone numbers.
About these practice questions
This Databricks-DE-Pro question is part of Courseiva's 267-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Pro practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Pro exam.