Databricks-Spark-Assoc Using Spark SQL Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  upper(tags)

  Why it's wrong here

  The upper function operates on a string, not an array. Applying it to an array column would result in an error or unexpected behavior. To uppercase each element in an array, you need a higher-order function like transform. Thus, this option is incorrect.

• ✓

  array_contains(tags, 'spark')

  Why this is correct

  The array_contains function checks if a given value exists in an array. It returns a boolean, which can be used in a filter condition. This directly addresses the requirement to filter rows where the array contains 'spark'. It is the standard function for this purpose in Spark SQL.

• ✓

  transform(tags, x -> upper(x))

  Why this is correct

  The transform function applies a lambda function to each element of an array, returning a new array. Using upper(x) converts each string to uppercase. This meets the requirement to transform the array to uppercase. It is a higher-order function available in Spark SQL 2.4+ and is efficient for such transformations.

• ✗

  filter(tags, x -> x = 'spark')

  Why it's wrong here

  The filter function returns a new array containing only elements that satisfy the predicate. While it can be used to check for presence by checking if the resulting array is not empty, it is not the direct function for filtering rows based on array containment. The requirement is to filter rows, not to filter elements within the array. Using array_contains is more straightforward and efficient.

• ✗

  explode(tags)

  Why it's wrong here

  The explode function creates a new row for each element in the array, which changes the granularity of the DataFrame. This is not suitable for filtering rows while preserving the original row structure. It would result in multiple rows per original row, complicating further transformations. Therefore, it is not appropriate for this scenario.