Exam objective 5.1 focuses on building solutions that extract hidden value from unstructured data. Knowledge mining with Azure Cognitive Search solves the problem of having mountains of documents, images, and text files but no easy way to ask specific questions or find specific information. For AI-102 candidates, this is the skill that turns a static archive into a dynamic, searchable knowledge base.
Jump to a section
A simple way to picture Knowledge Mining with Azure Cognitive Search
A senior museum archivist is tasked with helping visitors find any object in a vast, disorganised warehouse. The warehouse has thousands of items: paintings, sculptures, old letters, and artifacts, all piled in random boxes. The archivist's first job is to go through every single item, one by one, and write a detailed index card for each one. Each card notes the object's name, the material it's made from, the year it was made, its condition, and its exact shelf location. This process is the indexing.
Once all index cards are done, the archivist doesn't just leave them in a pile. They sort the cards into a custom filing cabinet with different sections: a section for 'red objects', a section for 'objects from the 1800s', and a section for 'fragile objects'. This sorting creates a map of the warehouse's contents, allowing a visitor to ask, 'Show me every red object from 1880 that is currently damaged,' and get a precise list of shelf numbers in seconds. The archivist's filing cabinet is the knowledge mining solution.
Without the archivist, a visitor would have to walk the aisles, open every box, and examine every object by hand — a process that could take weeks. The archivist's system transforms a chaotic warehouse into a searchable, structured library. Azure Cognitive Search does the same for digital documents: it indexes, organises, and enriches your data so you can ask precise questions and get instant answers.
Knowledge mining is the process of extracting useful information from data that is not already organised. In the real world, most data is 'unstructured': it is text in PDFs, captions in images, notes in scanned documents, or conversations in audio recordings. This data has valuable facts buried inside it, but a computer cannot easily answer a question like 'Show me all invoices from last year that mention a discount of more than 10 percent.'
Azure Cognitive Search is the Microsoft cloud service that performs knowledge mining. It is a Platform as a Service (PaaS) offering, meaning Microsoft manages the underlying servers, storage, and networking — you only configure the search service itself. The core job of Azure Cognitive Search is to take your raw, unstructured data and transform it into a structured, searchable index.
To understand how it works, you need to know three key components:
Data Sources: Where your raw data lives. This could be Azure Blob Storage (like a hard drive in the cloud), Azure SQL Database (a structured table), or even files in a shared folder. The search service connects to these sources to import data.
Indexers: Automated workers that pull data from the data source, process it, and push it into the search index. An indexer is like a pipeline: you tell it where to get the data, how to transform it, and where to put the results. Indexers can run on a schedule (e.g., every hour) or on demand.
The Index: A collection of structured fields that stores your processed data. Think of it as a very fast, customisable spreadsheet. Each piece of data becomes a 'document' in the index, with specific fields for specific types of information (e.g., a 'text', a 'date', a 'location'). Once data is in the index, you can query it using standard search syntax or more advanced AI-powered queries.
Where does the 'AI' part come in? Azure Cognitive Search can be enriched with AI skills from Azure Cognitive Services. For example, if you upload a scanned PDF of a handwritten contract, the search service can use Optical Character Recognition (OCR) to extract the text, then use a language model to detect key phrases like the contract date, parties involved, and monetary amounts. These AI enrichments create new fields in your index, making the hidden information searchable.
The entire process is called 'skillset enrichment'. A 'skillset' is a collection of AI skills (like OCR, entity recognition, or language detection) that you attach to an indexer. As the indexer imports documents, it runs them through each skill in the skillset, and the output becomes part of the index.
Why does this exist? Before knowledge mining, finding information in large volumes of documents was a manual, slow process. People had to read files one by one, or rely on simple keyword search that missed context (searching for 'train' would find 'train of thought' but not 'locomotive'). Knowledge mining replaces that with automated, intelligent indexing that understands meaning, not just words.
What does it replace? It replaces manual tagging, basic search engines that only match exact keywords, and the need for data scientists to manually label and structure data. It makes data accessible to any user through a simple search bar, API, or integrated application.
For AI-102, you need to know the three-step pipeline: Ingest (pull data from a source), Enrich (apply AI skills to extract insights), and Index (store the results in a searchable structure). This is the core workflow for any knowledge mining solution.
1. Create an Azure Cognitive Search Service
In the Azure Portal, you provision a new search service by selecting a pricing tier (Free, Basic, Standard). This creates the cloud endpoint that will host your index and handle queries. The service tier determines storage limits and throughput, so for production you often start with Standard.
2. Define the Index Schema
You design the index by specifying fields (e.g., 'Title', 'Content', 'Date', 'Location') and their data types (string, date, integer). You also set attributes per field, like whether it is searchable or filterable. This schema is the blueprint for your structured data.
3. Configure a Data Source
You point the search service to where your raw data lives. For example, you create a data source object that connects to an Azure Blob Storage container. The data source specifies the connection string and the container name.
4. Create a Skillset (Optional but Common)
You attach AI skills to enrich your data. For instance, you add an OCR skill to extract text from scanned images, then an Entity Recognition skill to identify names and dates. The skillset is a JSON document that lists each skill with its inputs and outputs.
5. Build and Run the Indexer
You create an indexer that connects the data source, the skillset (if any), and the index. The indexer pulls documents, runs them through the skillset, and writes the enriched output into the index fields. You run it initially to backload historical data, then set a schedule for incremental updates.
6. Query the Index and Test Results
After indexing completes, you test queries using the Azure Portal's Search Explorer or by sending HTTP requests to the REST API. You refine the search by adjusting scoring profiles, adding filters, or re-running the indexer with a modified schema.
Consider a medium-sized insurance company, 'SafeGuard Insurance', that has been in business for 40 years. They have millions of claim forms, policy documents, and customer correspondence stored as scanned PDFs and old Word files. Currently, if an adjuster needs to find all claims related to 'flood damage in Yorkshire from 2022', they must either remember exactly which folder it is in or search by file name, which is unlikely to be descriptive. The task could take hours or days.
SafeGuard decides to implement a knowledge mining solution using Azure Cognitive Search. Here is the step-by-step scenario:
Step 1: Data Preparation. The IT team moves all document files into Azure Blob Storage containers, organised loosely by year and type. They also configure a data source in Azure Cognitive Search that points to these containers.
Step 2: Index Design. They define an index with fields they want to search by: 'ClaimNumber', 'DateOfLoss', 'Location', 'ClaimAmount', 'PolicyHolderName', and 'Description'. These fields will be populated either from the original document metadata or from AI-powered extractions.
Step 3: Skillset Creation. They create a skillset with three skills: an OCR skill to extract text from scanned PDFs (many claim forms are handwritten), an entity recognition skill to identify locations (e.g., 'Yorkshire') and dates, and a key phrase extraction skill to pull out important terms like 'flood damage' or 'roof leak'. The OCR skill feeds its output text into the entity recognition and key phrase skills.
Step 4: Indexer Setup. They create an indexer that connects the data source, the skillset, and the index. They set it to run once to backload all historical data, then schedule it to run nightly to pick up new claims filed that day.
Step 5: Querying. Once the index is built, an adjuster opens a custom web app that queries the Azure Cognitive Search index. They type 'flood damage Yorkshire 2022'. The search service returns a list of claim documents, each with a highlighted snippet showing where the words were found. The results include not only exact matches but also documents where 'flood' was extracted as a key phrase even if the original text used 'water ingress'.
What does the IT professional do? They do not manually tag documents. Instead, they configure the Azure Cognitive Search service, write the skillset definition in JSON (a simple data format), test the indexer to ensure it processes a sample batch correctly, and then monitor the indexer's runs for errors. They also optimise the search by boosting certain fields (e.g., giving more weight to 'ClaimNumber' than 'Description') and adding scoring profiles to rank results by recency.
The outcome: the adjuster finds the relevant claims in seconds instead of hours. The company reduces claims processing time, improves customer satisfaction, and discovers trends (e.g., a spike in flooding claims in a specific region) that were previously invisible.
The AI-102 exam tests your ability to design and configure knowledge mining solutions, not to code them from scratch. You should expect 3-5 questions on this objective, often in the form of multiple-choice, drag-and-drop ordering, or case study scenarios. The exam is practical: it will give you a business requirement and ask you to choose the correct combination of components.
Key concepts the exam loves to test:
The three-step pipeline: Ingest, Enrich, Index. Be able to identify which step a given task falls into. For example, 'pulling data from Azure Blob Storage' is ingest. 'Using OCR to extract text' is enrich. 'Storing the output in a searchable field' is index.
Skillset components: You must know the built-in cognitive skills. The most tested ones are: OCR, Entity Recognition (for people, locations, organisations), Key Phrase Extraction, Language Detection, and Text Translation. The exam may ask which skill to use for a specific scenario. For example, 'extracting city names from a document' requires an Entity Recognition skill, not Key Phrase Extraction.
Data sources and indexers: Be clear on which data sources are supported (Azure Blob Storage, Azure SQL Database, Azure Cosmos DB, Azure Table Storage, and Azure Files) and how indexers connect them to an index. The exam may present a scenario with unsupported sources (like a local network drive) and expect you to know that data must first be moved to a supported Azure source.
Index fields and attributes: The exam tests index field attributes: 'searchable' (the field can be searched), 'filterable' (can be used in filters), 'sortable' (can be ordered), 'facetable' (can be used for drill-down navigation), and 'retrievable' (can be returned in search results). A common trap is asking which attribute to set for a field that should appear in search results but not be searchable itself (answer: retrievable = true, searchable = false).
Common traps and correct answer patterns:
Trap: 'Which component extracts text from images?' Trapped candidates choose 'indexer' instead of 'OCR skill'. The indexer orchestrates the pipeline; the OCR skill does the actual extraction.
Trap: 'You need to search documents by a specific date. Which field attribute should you set?' The answer is 'filterable', not 'sortable'. Filterable allows exact matching in a filter expression (e.g., $filter=date eq 2023-01-01). Sortable orders results by that field.
Trap: 'Which tool creates the search interface?' Azure Cognitive Search does not include a UI; it is a REST API. To build a search interface, you use a custom web app or a product like Azure Cognitive Search UI templates. The exam expects you to recognise that the search service itself is backend-only.
Trap: 'What happens when an indexer fails?' The indexer logs errors and continues processing other documents. It does not stop the entire run unless you explicitly configure it to fail on error. The exam may present a scenario where one document causes an error and ask whether the whole batch fails.
Memorise this: the skillset is the heart of knowledge mining. If the question asks about enriching data with AI, the answer will involve configuring a skillset attached to an indexer.
Knowledge mining with Azure Cognitive Search follows a three-step pipeline: Ingest (pull data), Enrich (apply AI skills), and Index (store structured results).
A skillset is a collection of AI skills (like OCR, Entity Recognition, Key Phrase Extraction) attached to an indexer to enrich data during indexing.
Azure Cognitive Search does not include a built-in user interface; you provide search results via a REST API that a custom web app or mobile interface consumes.
Index field attributes (searchable, filterable, sortable, facetable, retrievable) determine how each field behaves in queries and must be set during schema design.
An indexer can be scheduled to run periodically, automatically picking up new or modified documents from the data source without manual intervention.
Unstructured data (PDFs, images, text files) is the primary target for knowledge mining, as structured databases can already be queried with standard SQL.
These come up on the exam all the time. Here's how to tell them apart.
Azure Cognitive Search Index
Designed for full-text search and fuzzy matching
Fields are defined with attributes like searchable and facetable
Data is ingested via indexers from unstructured sources
SQL Database Table
Designed for exact queries and transactional operations
Columns have strict data types with constraints like primary keys
Data is inserted via SQL INSERT statements from structured applications
Skillset (Enrichment Pipeline)
Performs AI operations like OCR and entity recognition
Outputs enriched data that becomes new fields in the index
Configured as a JSON array of skill definitions
Indexer (Data Ingestion Pipeline)
Orchestrates the overall import process from data source to index
Calls the skillset during processing but does not perform AI itself
Configured with data source details, schedule, and error handling
Built-in Cognitive Skills
Pre-built by Microsoft and require no coding
Cover common tasks: OCR, language detection, entity extraction
Billed through an attached Azure AI Services account
Custom Skills
Custom code (e.g., Python Azure Function) you write and host
Used for specialised tasks not covered by built-in skills
Run in your own compute environment and billed separately
Mistake
Azure Cognitive Search is just a cloud search engine that works like Google for your files.
Correct
It is a full knowledge mining solution that indexes your data, enriches it with AI, and structures it into custom fields for precise querying.
Beginners often think 'search equals Google', but Azure Cognitive Search requires upfront configuration of data sources, index schemas, and skillsets — it is not a point-and-click instant search tool.
Mistake
You need to write code to use Azure Cognitive Search.
Correct
The service is largely configured through a graphical portal (Azure Portal) or by writing JSON definitions. No programming knowledge is required for basic setup, though coding (e.g., in C# or Python) helps for advanced customisation.
Many non-developers assume cloud AI tools require heavy coding, but Azure Cognitive Search is designed to be configurable by IT administrators and data analysts who are not software engineers.
Mistake
The indexer automatically creates the index schema for you.
Correct
You must define the index schema (the fields and their data types) manually before or during the indexer setup. The indexer populates the fields but does not design them automatically.
Beginners expect a 'magical' tool that analyses data and builds a perfect schema. In reality, the user must specify which fields they need (e.g., 'ClaimNumber', 'Date') and map them to incoming data.
Mistake
Cognitive Skills can only be applied to images.
Correct
Cognitive Skills can be applied to text, images, and even audio files (transcription via Speech-to-Text is a separate skill). OCR works on images, but Entity Recognition works on text extracted from any source.
The buzzword 'AI' often conjures images of object detection in photos, but structured text enrichment (e.g., extracting phone numbers from a PDF) is equally common and tested on the exam.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
They are the same service. 'Azure Cognitive Search' was the original name, and it was recently updated to 'Azure AI Search' to reflect its AI capabilities. For the AI-102 exam, both names refer to the same product.
No, a single index is tied to one data source (e.g., one Azure Blob Storage container). To combine data from multiple sources, you would either push data from each source into a single index using a custom pipeline or create multiple indexes and merge results in your application.
Yes, to use built-in AI skills (OCR, Entity Recognition, etc.), you must attach an Azure Cognitive Services (now Azure AI Services) multi-service account to your search service. This provides the API keys and billing for the AI operations.
The indexer will log an error for that specific file and continue processing the remaining ones. It does not halt the entire indexing run unless you explicitly configure it to stop on failure.
Modifying an existing index schema is very limited. You cannot add new fields to an existing index that already contains data unless you recreate the index or use aliases. It is best to design the schema thoroughly before the first indexing run.
No, it works with images (via OCR), audio files (via transcription skills), and even videos (by extracting frames as images). However, the AI skills must be configured for each media type.
You've finished Knowledge Mining with Azure Cognitive Search. Continue through the AI-102 study guide to build a complete picture of the exam.
Done with this chapter?