Databricks-Spark-Assoc Developing DataFrame/DataSet API Applications Practice Question
A data engineer has a PySpark DataFrame `events` with columns `event_ts` (TimestampType) and `user_id`. They must produce a new DataFrame where each row shows the event and the timestamp of that same user's previous event, ordered by `event_ts` within each `user_id`. Which code snippet correctly accomplishes this?
⚠ Common exam trap
The trap here is assuming that `orderBy` on a DataFrame establishes ordering for analytic functions, when ordering must instead be declared inside the `Window` specification.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
events.withColumn("prev_ts", lag("event_ts").over(Window.partitionBy("user_id").orderBy("event_ts")))
A window function with `partitionBy("user_id").orderBy("event_ts")` defines a frame scoped to each user in chronological order, and `lag` reads the value one row back inside that frame. That yields the same user's prior event timestamp on every row, returning null only for each user's earliest event, which matches the requested output exactly.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
events.withColumn("prev_ts", lag("event_ts").over(Window.partitionBy("user_id").orderBy("event_ts")))
Why this is correct
`lag` is an analytic window function that returns the value from the preceding row within the window frame. Partitioning by `user_id` and ordering by `event_ts` makes the previous row the same user's chronologically earlier event, exactly the required semantics, and `lag` returns null for the first event of each user without dropping the row.
- ✗
events.withColumn("prev_ts", lead("event_ts").over(Window.orderBy("event_ts")))
Why it's wrong here
`lead` looks forward to the following row, giving the user's next event rather than the previous one, which inverts the requirement. The window also lacks `partitionBy("user_id")`, so ordering spans the entire DataFrame and the "next" row may belong to a completely different user, producing incorrect cross-user values at partition boundaries.
- ✗
events.groupBy("user_id").agg(max("event_ts").alias("prev_ts"))
Why it's wrong here
Aggregating with `max` collapses all rows for a user into a single row, so the per-event output shape is destroyed and only the latest timestamp survives. It cannot attach a distinct prior timestamp to every individual event, and it never produces null for a user's first event. This changes both the grain and the meaning of the result.
- ✗
events.orderBy("event_ts").withColumn("prev_ts", first("event_ts"))
Why it's wrong here
`orderBy` only affects output row ordering and does not establish a window frame, so `first` without an `over` specification is not a valid analytic expression in this context. Even if it parsed, `first` would yield a single global value, not the immediately preceding timestamp per user, and no partition boundary is defined.
About these practice questions
This Databricks-Spark-Assoc question is part of Courseiva's 295-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.