You are processing a DataFrame containing nested JSON structures using Spark SQL functions. Which TWO of the following approaches allow you to flatten or extract fields from a struct column named 'user_info'? (Choose two)
You are joining two large DataFrames, 'sales' and 'products', on a common key 'product_id'. To ensure the join operation executes efficiently without causing driver memory issues, what is the best practice approach in Spark?
A data engineer needs to aggregate sales data by store region and compute the running total of sales ordered chronologically within each region using window functions. Which DataFrame API pattern correctly accomplishes this without causing an out-of-memory error on the driver?
An engineering team needs to process a large DataFrame containing user transaction logs. They must filter rows where the amount exceeds 500, and then extract the top 10 highest amounts without using global sorting. Which transformation sequence achieves this efficiently in PySpark?
A data engineer writes a PySpark application that reads a JSON file containing nested structures and arrays. They need to flatten the arrays into separate rows while preserving other field values. Which TWO DataFrame transformations should they use together in their pipeline? (Choose two)
You have a PySpark DataFrame named df containing employee records with columns department, name, and salary. You need to calculate the running total of salaries for each department ordered by salary in ascending order, without reducing the number of output rows. Which DataFrame transformation should you apply?
A developer is building a PySpark job that reads a Parquet file into a DataFrame `events` and needs to add a monotonically increasing, unique ID column to each row for downstream deduplication. The DataFrame has no existing numeric column that guarantees uniqueness, and the job runs on a cluster with multiple executors. Which single DataFrame API call should they use to attach such an ID column without triggering a full shuffle of the existing data?
A data engineer has a PySpark DataFrame `df` with columns `order_id`, `customer_id`, and `order_ts` (a TimestampType column). The engineer must return only the rows where `order_ts` falls on or after January 1, 2023, and must keep the result as a DataFrame without triggering collection to the driver. Which transformation should be used?
A developer is working with a DataFrame `df` that has a `status` string column containing values like 'active', 'inactive', and 'pending'. They need to replace every occurrence of 'inactive' with 'disabled' in that column while keeping all other values unchanged. Which operation should they use?
A data engineer is building a PySpark application that reads a stream of JSON events from a Kafka topic into a DataFrame. The events have a nested struct field `device` containing `os` and `version`. The engineer needs to add a flattened column `os_version` that concatenates `device.os` and `device.version` with a hyphen, while keeping all original columns. Which code snippet correctly accomplishes this?
A data engineer has a PySpark DataFrame `df` with columns `event_id`, `user_id`, and `payload` (a StringType column holding JSON text). The engineer must parse `payload` into a struct column exposing fields `action` and `duration`. Which TWO approaches correctly produce a struct column from the JSON string? (Choose two.)
A data engineer has a streaming DataFrame `events` from a Kafka source with columns `user_id` and `event_time`, and must compute a per-user count of events that updates continuously as new micro-batches arrive. Which TWO operations or clauses are required to produce this continuously updated result? (Choose two.)
A developer has a PySpark DataFrame `events` with a `timestamp` column stored as a string in ISO-8601 format (for example, `2024-03-15T08:45:30Z`). They need to extract only the calendar date (year-month-day) into a new column `event_date` of type DateType, without changing the original `timestamp` column. Which single expression accomplishes this?
A data engineer is working with a DataFrame `df` that has a column `event_time` of type Timestamp and a column `user_id`. The engineer needs to create a new column `session_id` that increments by 1 whenever the gap between consecutive `event_time` values for the same `user_id` exceeds 30 minutes. The data is sorted by `user_id` and `event_time`. Which approach correctly implements this sessionization logic?
A developer must remove duplicate rows from a DataFrame `orders` so that only the most recent order per `customer_id` is kept, based on `order_ts`. Which TWO approaches achieve this correctly? (Choose two.)
A developer must persist a DataFrame `clean_df` to Parquet while partitioning the output on disk by the `region` column and controlling the number of output files per partition directory. Which TWO statements about using `clean_df.write` are correct? (Choose two.)
A data engineer is working with a PySpark DataFrame `df` that contains a column `tags` of type ArrayType(StringType). The engineer needs to create a new column `tag_count` that contains the number of elements in each array, and also filter out rows where the array is empty or null. Which TWO of the following approaches correctly achieve both the transformation and filtering? (Choose two.)
A data engineer has a DataFrame `df` with columns `product_id` and `price`. They need to create a new DataFrame that contains only the `product_id` and a new column `price_with_tax` calculated as `price * 1.1`. Which code snippet correctly produces this result?
A developer has a DataFrame `df` with a nested struct column `address` containing fields `city`, `state`, and `zip`. They need to expose `city` and `zip` as top-level columns. Which TWO approaches correctly achieve this using the DataFrame API? (Choose two.)
A data engineer is using a PySpark DataFrame `df` with a column `category` and a column `value`. They need to compute the percentage of total `value` for each `category`, rounded to two decimal places. The total sum of `value` across all rows must be computed over the entire DataFrame, not per category. Which code snippet correctly achieves this?
A data engineer has a PySpark DataFrame `events` with a `timestamp` column stored as a string in the format `yyyy-MM-dd HH:mm:ss`. They need to add a new column `event_date` that contains only the date part as a `date` type, and they want to avoid any implicit string-to-date casting that could produce nulls on malformed rows. Which approach correctly creates `event_date` while preserving a strict parse behavior?
A developer is working with a PySpark DataFrame `transactions` that has columns `account_id`, `txn_date` (DateType), and `amount` (DoubleType). They must produce a new DataFrame containing only the rows where `amount` is strictly greater than 500.00 and `txn_date` is on or after 2024-01-01. The predicate must be pushed down to the underlying Parquet scan to minimize I/O. Which code snippet accomplishes this?
You have a large DataFrame containing user transaction logs. You need to read this data and immediately repartition it by user_id to optimize downstream filtering operations. Which DataFrame API method should you use?
You need to add a new calculated column 'discounted_price' to an existing DataFrame 'df' by multiplying 'price' by 0.9. Which DataFrame transformation accomplishes this correctly?
A data engineer needs to join two large DataFrames, `sales` and `products`, on the `product_id` column. The `products` DataFrame is extremely small and fits entirely in a single executor's memory. To optimize performance and avoid a costly shuffle join across the network, which strategy should be applied using the Spark DataFrame API?
You are processing a large, highly skewed PySpark DataFrame in Databricks and want to optimize a forthcoming join operation against a small lookup dimension table. Which TWO strategies are valid and effective DataFrame API techniques to optimize this join performance? (Choose TWO)
A data engineer has a PySpark DataFrame `events` with columns `user_id`, `event_time` (timestamp), and `payload` (string). They must produce a new DataFrame where each row is enriched with the `payload` value from the user's immediately preceding event, ordered by `event_time`, without collapsing any rows. Which TWO approaches accomplish this? (Choose two.)
A developer has a PySpark DataFrame `df` with columns `order_id`, `customer_id`, and `order_ts` (timestamp). They need to return only the most recent order per customer, keeping all original columns, and they want to avoid a self-join or a manual sort-then-dropDuplicates approach. Which DataFrame operation should they use?
A developer is building a PySpark job that reads a Parquet dataset with 2,000 files into `df`. They call `df.cache()` and then execute three separate actions in the same session. The Spark UI shows the Parquet files are read from storage three times, and the cache never appears in the Storage tab. What is the most likely cause?
A developer has a DataFrame `events` with columns `user_id`, `event_type`, and `payload` (a JSON string). They need to extract the `device` field from `payload` into a new column without changing the other columns or the row count. Which approach is correct?
A data engineer has a PySpark DataFrame `orders` with a string column `order_ts` formatted as `yyyy-MM-dd HH:mm:ss`. They need a new column `order_date` containing only the date portion as a `date` type, so downstream code can filter by day. Which approach is correct?
A data engineer has a PySpark DataFrame `events` with columns `event_ts` (TimestampType) and `user_id`. They must produce a new DataFrame where each row shows the event and the timestamp of that same user's previous event, ordered by `event_ts` within each `user_id`. Which code snippet correctly accomplishes this?
A data engineer has a PySpark DataFrame `readings` with columns `sensor_id` (string) and `celsius` (double). The engineer must produce a new DataFrame where every temperature is converted to Fahrenheit using the formula `celsius * 9/5 + 32`, while keeping both the original `sensor_id` and a column named `fahrenheit`, and must avoid collecting data to the driver. Which DataFrame operation should be used?
A developer must join a 4 TB `transactions` DataFrame against a 900 MB `merchants` DataFrame on `merchant_id`. The cluster has 40 executors each with 16 GB of memory, and the job currently shuffles the large side. They want to avoid the shuffle entirely. Which change should they make?
A developer has a DataFrame `raw` and needs to permanently persist it as Parquet partitioned by `region`, overwriting any existing data at that path, without registering it in the metastore. Which call achieves this?
A developer needs to join `orders` (large) with `customers` (small, a few thousand rows) on `customer_id`. They want to broadcast the small table and confirm the broadcast actually took effect. Which combination of actions is correct?
A developer needs to add a column `full_name` to DataFrame `people` by concatenating `first_name` and `last_name` with a single space, and must handle rows where `last_name` is null by producing just the first name. Which expression is correct?
A data engineer has a DataFrame `events` with columns `user_id` and `ts`. They need to add a column `prev_ts` that holds the previous event timestamp for each user, ordered by `ts` ascending, without collapsing rows. Which operation accomplishes this?
A data engineer has a batch DataFrame `df` with a `status` string column and wants to keep only rows where `status` equals "active". The engineer wants the filter applied as early as possible in the plan and does not want a shuffle. Which operation best fits this requirement?
A data engineer is writing a PySpark job that must read Parquet data, apply transformations, and then write the result. They want to avoid recomputation if the resulting DataFrame is referenced multiple times in later stages. Which action best accomplishes this?
A data engineer has a DataFrame `orders` with columns `order_id`, `customer_id`, and `amount`, and a small lookup DataFrame `tiers` with `customer_id` and `tier`. The engineer wants to attach the tier to every order. Some orders have a customer_id that is not present in tiers, and those orders must still appear with a null tier. Which operation produces this result?
A data engineer has two DataFrames: `orders` with columns `order_id` and `customer_id`, and `customers` with columns `customer_id` and `region`. They must produce a result containing only orders whose `customer_id` exists in `customers`, keeping every matching order row exactly once with no customer columns added. Which operation should be used?
A data engineer is building a feature pipeline and needs to add a monotonically increasing integer column `row_num` to a DataFrame `df` that assigns consecutive numbers to rows within each partition of a specified ordering, similar to a window function. Which approach uses the DataFrame API to compute this value?
A developer has a DataFrame `df` with columns `id` and `score`, and several rows contain null values in `score`. They want a new DataFrame in which rows with a null `score` are removed, keeping only rows where `score` is present. Which single call achieves this?
A developer is building a feature that must, for each `customer_id`, concatenate the distinct `product` values from many rows into a single comma-separated string. The result must contain each product only once per customer. Which single approach produces this result?
A data engineer has a DataFrame `df` with columns `order_id`, `customer_id`, and `order_total`. They need to create a new DataFrame containing only the rows where `order_total` is greater than 100 and only the columns `order_id` and `order_total`. Which combination of DataFrame operations accomplishes this most efficiently?
A developer has two DataFrames, `orders` (columns `order_id`, `customer_id`) and `customers` (columns `customer_id`, `customer_name`). They need a result containing every order, with the matching customer name where one exists and null where the customer is not found. Which single join configuration guarantees this?
A developer has a DataFrame `trades` with columns `trade_id`, `symbol`, and `price`. They want to add a column `prev_price` containing the price of the immediately preceding trade for the same symbol, ordered by `trade_id` ascending, without collapsing rows. Which transformation should they use?
A developer must combine two DataFrames, `left_df` and `right_df`, on a key column `id`. They need every row from `left_df` regardless of whether a match exists in `right_df`, and matching rows from `right_df` where available, with unmatched right-side columns filled as null. Which join invocation produces this result?
A developer has a DataFrame `df` and wants to remove duplicate rows considering only the columns `user_id` and `event_type`, keeping the first occurrence according to the current row order. Which DataFrame operation achieves this?
A data engineer needs to add a column `rank_in_dept` to a DataFrame `employees` that ranks each employee by `salary` descending within their `department`, but they must not collapse rows. They also want ties to receive the same rank with gaps afterward. Which expression correctly produces this column using the DataFrame API?
A developer is writing a PySpark job that must read a Parquet dataset, apply several transformations, and then write the result back to storage. They want to ensure the schema of the written data is inferred directly from the DataFrame rather than from any external definition, and they want to append to an existing Parquet directory. Which write configuration accomplishes this?
A data engineer is building a PySpark application that must validate incoming records in a DataFrame `raw` before loading them into a curated table. They want to apply user-defined validation logic that cannot be expressed with built-in functions, and they want the result to remain a DataFrame column of Boolean values. Which TWO approaches allow this? (Choose two.)