Azure AI Search for RAG Applications

Azure AI Search (formerly Azure Cognitive Search) is a cloud search-as-a-service solution that provides full-text search, vector search, and hybrid search capabilities. In the context of RAG, it serves as the retrieval engine that supplies relevant documents or text chunks to an LLM. The LLM then uses these retrieved pieces to generate a grounded answer. Without a retrieval system, an LLM relies solely on its training data, which may be outdated or lack domain-specific information. RAG solves this by fetching fresh, relevant content from a search index at inference time.

The process has two main phases: indexing and querying.

Indexing Phase: 1. Data Ingestion: Documents (PDFs, Word, HTML, etc.) are loaded into Azure AI Search via a data source connector (e.g., Azure Blob Storage, Cosmos DB). 2. Document Cracking: The service parses documents into text and metadata. For PDFs, it extracts text and preserves structure (headings, tables). 3. Text Splitting (Chunking): Documents are split into smaller chunks, typically 500-1000 tokens, with overlap (e.g., 100 tokens) to preserve context across chunk boundaries. This is often done using Azure AI Document Intelligence or custom logic. 4. Vectorization: Each chunk is converted into a vector embedding using a text-embedding model (e.g., Azure OpenAI's text-embedding-ada-002). The embedding is a high-dimensional vector (1536 dimensions for ada-002) that captures semantic meaning. 5. Index Creation: The chunks, their vectors, and metadata are stored in the search index. The index includes fields like id, content, content_vector, source_file, page_number. The content_vector field is configured with a vector search algorithm (e.g., HNSW – Hierarchical Navigable Small World) for efficient similarity search.

Querying Phase: 1. User Query: The user submits a question (e.g., "What is the expense reimbursement policy?"). 2. Query Vectorization: The query is converted into a vector using the same embedding model used during indexing. 3. Search Execution: Azure AI Search performs a hybrid search: - Keyword search: Uses BM25 ranking to find chunks containing the exact words "expense," "reimbursement," "policy." - Vector search: Uses cosine similarity to find chunks with vectors close to the query vector. - Hybrid fusion: Combines results using Reciprocal Rank Fusion (RRF) to produce a unified ranked list. 4. Result Retrieval: The top K results (e.g., 5) are returned, each containing the chunk text and metadata. 5. LLM Generation: The chunks are injected into a prompt template (e.g., "Context: {chunk1} {chunk2} ... Question: {query} Answer:") and sent to an LLM (e.g., GPT-4). The LLM generates an answer grounded in the provided context.

You can configure Azure AI Search via the Azure portal, REST API, or Azure CLI. Example CLI commands:

# Create a search service
az search service create --name myservice --resource-group myrg --sku basic

# Create an index
az search index create --service-name myservice --name myindex --fields @fields.json

# Create a data source
az search datasource create --service-name myservice --name myblob --type azureblob --credentials "connectionString=..." --container mycontainer

# Create an indexer
az search indexer create --service-name myservice --name myindexer --data-source-name myblob --target-index-name myindex

# Run an indexer
az search indexer run --service-name myservice --name myindexer

To verify indexing status:

az search indexer show --service-name myservice --name myindexer --query status

A large insurance company uses Azure AI Search to power a RAG-based customer support chatbot. They have thousands of PDF policy documents and claim-handling guides. The problem: agents spent minutes searching for answers, and the previous chatbot hallucinated incorrect policy details. Solution: They ingested all documents into Azure Blob Storage, created an index with vector embeddings from text-embedding-ada-002, and set up an indexer to run nightly. The chatbot's LLM (GPT-4) is prompted with the top 5 retrieved chunks. Results: First-call resolution improved by 40%, and hallucination rates dropped below 1%. Common pitfall: Not chunking properly—if chunks are too large (e.g., entire 50-page document), the LLM context window fills with irrelevant text, degrading accuracy. They optimized chunk size to 512 tokens with 100-token overlap.

A multinational corporation deployed a RAG assistant for employees to query HR policies (benefits, leave, code of conduct). They used Azure AI Search with hybrid search because employees often used different terminology (e.g., "vacation" vs. "annual leave"). The vector search captured semantic similarity, while keyword search ensured exact matches for policy numbers. They configured semantic search re-ranking to improve relevance. Scale: 50,000 documents, 1 million chunks, S2 tier. Performance: Average query latency under 500 ms. Misconfiguration: Initially, they used only keyword search, which missed semantically related queries. After enabling hybrid search, accuracy improved by 60%.

A law firm uses RAG to help lawyers quickly find precedents in a corpus of 500,000 legal documents. They needed high precision because missing a relevant case could have serious consequences. They used Azure AI Search with vector search only (no keyword) because legal language is highly specific and exact word matches often miss synonyms. They tuned HNSW parameters: M=32, efConstruction=400, efSearch=800. They also used metadata filtering (e.g., court, date range) to narrow results. Pitfall: Without proper access control, sensitive documents were exposed to all users. They implemented index-level security using Azure AD and field-level security filters.