Databricks-Spark-Assoc Using Spark SQL Practice Question
A developer has a Spark SQL DataFrame `df` with an array column named `scores` containing integers. They need to create a new column `passing` that is true only when every element in `scores` is greater than or equal to 70. Which Spark SQL higher-order function should they use?
⚠ Common exam trap
Many candidates confuse higher-order functions that return arrays (transform, filter) with those that return booleans (exists, forall), and mixing up exists (any) with forall (all).
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
forall(scores, s -> s >= 70)
The forall higher-order function is designed to test whether every element in an array satisfies a given predicate, returning a single boolean. In this scenario, it correctly produces a column that is true only when all scores are 70 or higher. transform maps each element, filter selects elements, and exists checks for at least one match, none of which yield the required all-elements condition.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
exists(scores, s -> s >= 70)
Why it's wrong here
The exists higher-order function returns true if at least one element satisfies the predicate. It would return true if any score is 70 or above, not if all are. This is the logical OR across elements, whereas the requirement is a logical AND across elements. Using exists would incorrectly mark rows as passing when only some scores meet the threshold.
- ✗
filter(scores, s -> s >= 70)
Why it's wrong here
The filter higher-order function returns a new array containing only the elements that satisfy the predicate. It does not return a boolean; it would give an array of the passing scores. To determine if all elements pass, you would need to compare the size of the filtered array to the original, which is not what the question asks for.
- ✓
forall(scores, s -> s >= 70)
Why this is correct
The forall higher-order function returns true only if the lambda predicate evaluates to true for every element in the array. In this scenario, forall(scores, s -> s >= 70) yields a single boolean column that is true exactly when all scores are at least 70, which matches the requirement. It short-circuits on the first false, making it efficient.
- ✗
transform(scores, s -> s >= 70)
Why it's wrong here
The transform function applies a lambda to each element and returns a new array of the same length, so it would produce an array of booleans, not a single boolean. To get a single true or false, you need a reduction-style higher-order function like exists or forall. transform is for element-wise mapping, not for aggregating an array into a scalar condition.
About these practice questions
One of 295 original Databricks-Spark-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.