A company stores application logs in Amazon S3 and wants to analyze them using Amazon Athena. The logs are in JSON format and are compressed with gzip. The data engineer needs to create an Athena table that can query these logs efficiently. The logs are stored in an S3 bucket with the prefix logs/year=2023/month=10/day=15/. The engineer wants to minimize query costs and improve performance. Which action should the engineer take?
Creating an external table with the JSON SerDe and defining partitions allows Athena to read the JSON logs and use partition pruning to scan only relevant data. Specifying the root S3 location and adding partitions (either manually or via MSCK REPAIR) enables efficient queries and reduces cost by limiting data scanned.
Why this answer
To query JSON logs efficiently in Athena, the table must use the JSON SerDe to parse the data correctly. Defining partitions on year, month, and day allows Athena to prune partitions based on query filters, reducing the amount of data scanned and lowering costs. The table location should point to the root prefix so that all partitions are included.
This combination provides accurate parsing and cost-effective queries.
Exam trap
The trap here is selecting a SerDe based on compression or performance assumptions rather than matching it to the actual data format, which is JSON.