A data engineer is building a Silver table in Delta Lake from a Bronze table that contains raw JSON events. The engineer needs to flatten a nested struct column named 'device' with fields 'type' and 'os', and also extract a field from an array of structs named 'events'. The goal is to produce a clean, denormalized Silver table. Which PySpark operation should the engineer use to achieve this transformation efficiently?
This approach flattens the 'device' struct by directly selecting its fields as separate columns, and uses explode() on the 'events' array to create a new row for each event, then selects the 'event_name' field from the exploded struct. This results in a denormalized table where each row represents an event with associated device information, which is a common pattern for Silver tables.
Why this answer
To flatten a struct, you select its fields directly. To flatten an array of structs, you explode the array to create multiple rows, then select the desired fields from the exploded struct. Combining these operations yields a denormalized table suitable for Silver.
The other options either fail to flatten the struct, incorrectly use array functions, or do not fully denormalize the data.
Exam trap
The trap here is confusing explode() with other array functions like flatten(), or assuming that dot notation alone flattens nested structures.