Azure AI Search for Unstructured Data

Azure AI Search (formerly Azure Cognitive Search) is a fully managed, cloud-based search-as-a-service solution that enables developers to build rich search experiences over heterogeneous content. It is designed primarily for unstructured and semi-structured data—documents, images, text files, web pages, emails, and other content that does not fit neatly into a relational schema. The service ingests data from various sources, indexes it using inverted indexes and AI-powered enrichment, and exposes search capabilities via REST APIs or SDKs.

The core problem Azure AI Search solves is the difficulty of searching across large volumes of unstructured data efficiently. Traditional databases excel at querying structured data (e.g., SELECT * FROM Customers WHERE LastName = 'Smith'), but they struggle with free-text queries, relevance ranking, and handling content like PDFs or images. Azure AI Search fills this gap by providing:

The process of making unstructured data searchable in Azure AI Search occurs in three main phases: indexing, enrichment (optional), and querying.

#### 1. Indexing Indexing is the process of extracting data from a source, transforming it, and loading it into a search index. The key components are:

#### 2. AI Enrichment (Skillsets) AI enrichment is optional but powerful for unstructured data. It uses a skillset—a pipeline of cognitive skills that transform or extract information from the raw content. Skillsets are attached to indexers and execute during indexing. Each skill takes input from the document or previous skills and produces output that is written to the index.

Built-in skills include: - OCR Skill: Extracts text from images (JPEG, PNG, TIFF, BMP) using optical character recognition. Supports multiple languages. - Key Phrase Extraction Skill: Identifies key talking points from text (e.g., from a PDF). - Entity Recognition Skill: Identifies named entities (people, organizations, locations, dates, etc.). - Language Detection Skill: Detects the language of text (supports 120+ languages). - Sentiment Analysis Skill: Scores text sentiment from 0 (negative) to 1 (positive). - Merge Skill: Merges fields together (e.g., concatenate title and content). - Conditional Skill: Implements if-then-else logic. - Custom Skill: Allows you to call an Azure Function or web API for custom processing.

Skillsets are defined in JSON. Here is an example snippet:

{
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Text.KeyPhraseExtractionSkill",
      "name": "keyphrase",
      "context": "/document",
      "inputs": [
        { "name": "text", "source": "/document/content" }
      ],
      "outputs": [
        { "name": "keyPhrases", "targetName": "myKeyPhrases" }
      ]
    }
  ]
}

#### 3. Querying Once the index is populated, clients can query it via REST API or SDK. Queries are HTTP POST requests to the search endpoint with parameters like search, filter, orderby, facets, and highlight. The search engine parses the query, looks up terms in the inverted index, computes relevance scores (BM25), and returns results ordered by score.

Example query:

POST https://[service-name].search.windows.net/indexes/[index-name]/docs/search?api-version=2020-06-30
Content-Type: application/json
{
  "search": "Azure AI Search",
  "queryType": "simple",
  "searchMode": "any",
  "top": 10
}

To create an indexer from the Azure CLI:

az search indexer create --name myindexer --service-name myservice --data-source-name mydatasource --target-index-name myindex --skillset-name myskillset --schedule "interval" PT5M

To check indexer status:

az search indexer show-status --name myindexer --service-name myservice

To run an indexer on demand:

az search indexer run --name myindexer --service-name myservice

Azure AI Search integrates with many Azure services: - Azure Blob Storage: Primary source for unstructured data (PDFs, images, text files). Indexers can extract metadata and content from blobs. - Azure Cognitive Services: Skillsets use Cognitive Services APIs (e.g., Computer Vision for OCR, Text Analytics for key phrases). You must attach a Cognitive Services resource to enable billable skills. - Azure Data Factory: Can orchestrate data movement into Blob Storage before indexing. - Azure Synapse Analytics: Can use search indexes as a source for enrichment pipelines. - Power BI: Can connect to search indexes via the Search Index connector for reporting. - Azure Functions: Custom skills can call Azure Functions for domain-specific processing. - Azure Monitor: Logs and metrics for indexing operations, query latency, and errors.

The service is not a replacement for Azure Cosmos DB (OLTP) or Azure SQL Database (relational queries); it is complementary for search-centric workloads.