Databricks-Spark-Assoc Developing DataFrame/DataSet API Applications Practice Question
A developer has a PySpark DataFrame `df` with columns `order_id`, `customer_id`, and `order_ts` (timestamp). They need to return only the most recent order per customer, keeping all original columns, and they want to avoid a self-join or a manual sort-then-dropDuplicates approach. Which DataFrame operation should they use?
⚠ Common exam trap
The trap here is assuming that `dropDuplicates` on a grouping column keeps the latest record, when in fact it keeps an arbitrary row and ignores any ordering.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use a Window partitioned by `customer_id` ordered by `order_ts` descending, add `row_number()`, then filter for row number equal to 1
Selecting the latest row per group while preserving all columns is exactly what a Window with `row_number()` solves. Partitioning by the grouping key and ordering by the timestamp descending makes the most recent record rank 1, and filtering on that rank returns complete rows. Aggregations drop columns, global limits collapse groups, and dropDuplicates cannot express ordering.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
df.dropDuplicates(["customer_id"])
Why it's wrong here
`dropDuplicates` on `customer_id` keeps an arbitrary row per customer based on internal partition ordering, not the latest `order_ts`. The chosen row is non-deterministic and will not reliably be the most recent order. It also does not accept an ordering criterion, so it cannot express 'most recent' semantics at all.
- ✗
df.groupBy("customer_id").max("order_ts")
Why it's wrong here
This aggregation returns only `customer_id` and the maximum timestamp, dropping `order_id` and any other columns. It satisfies the 'most recent timestamp per customer' requirement, but it does not preserve the full original row, so the developer would lose the order identifier and any additional attributes. It also forces a shuffle-based aggregation rather than the intended windowed row selection.
- ✓
Use a Window partitioned by `customer_id` ordered by `order_ts` descending, add `row_number()`, then filter for row number equal to 1
Why this is correct
A Window partitioned by `customer_id` and ordered by `order_ts` descending assigns rank 1 to the latest order per customer. Adding `row_number().over(window)` and filtering `row_number == 1` returns the full original row, preserving all columns. This is the idiomatic PySpark replacement for a self-join or sort-then-dedup pattern and scales with partitioning.
- ✗
df.orderBy("order_ts", ascending=False).limit(1)
Why it's wrong here
This returns only the single most recent order across the entire DataFrame, not one row per customer. The `limit(1)` collapses the result to a single record regardless of how many distinct customers exist. It also performs a global sort, which is expensive and unnecessary when the goal is per-group selection.
About these practice questions
This Databricks-Spark-Assoc question is part of Courseiva's 295-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.