Databricks-DE-Pro Data Ingestion and Acquisition Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  cloudFiles.schemaLocation

  Why it's wrong here

  The cloudFiles.schemaLocation parameter specifies a location to store the inferred schema and metadata for the stream. It is used for schema evolution and checkpointing, not for filtering files. Setting this parameter would not affect which files are ingested; it only manages schema information and does not restrict file selection based on extension.

• ✗

  cloudFiles.includeExistingFiles

  Why it's wrong here

  This parameter controls whether Auto Loader processes files that already exist in the source location when the stream starts. It does not filter by file extension. Setting it to true would ingest all existing files regardless of format, and setting it to false would ignore existing files. It is unrelated to selecting only .json files and would not prevent .json.gz files from being ingested.

• ✗

  cloudFiles.format

  Why it's wrong here

  The cloudFiles.format parameter specifies the format of the source files, such as json, csv, or parquet. It tells Auto Loader how to parse the file contents, but it does not filter which files are picked up based on their extension. All files in the directory would still be considered, and .json.gz files would also be read as JSON, potentially causing issues.

• ✓

  cloudFiles.pathGlobFilter

  Why this is correct

  The cloudFiles.pathGlobFilter parameter allows you to specify a glob pattern to filter files based on their path or extension. By setting it to '*.json', Auto Loader will only process files ending with .json, excluding .json.gz files. This is the correct way to selectively ingest files by extension in Auto Loader, ensuring only the desired files are read.