Named Entity Recognition (NER)

Named Entity Recognition (NER) is a natural language processing (NLP) technique that identifies and classifies named entities in unstructured text into predefined categories. In Azure AI Language, NER is a prebuilt model that can recognize entities across multiple categories, including Person, Organization, Location, DateTime, Quantity, Event, Product, and more. The service is part of the Azure AI Language (formerly Text Analytics) API, which is a managed cognitive service. NER is used to extract structured information from unstructured text, enabling downstream tasks like information retrieval, knowledge graph construction, and content summarization.

Azure AI Language's NER model uses a combination of deep learning (transformer-based models) and pattern matching. The process involves several steps:

4. Entity Extraction: The tagged tokens are grouped into complete entities. For each entity, the model outputs: - Text: The exact string as it appears in the input. - Category: The entity type (e.g., Person, Organization). - Subcategory: A more specific type (e.g., for Location, subcategories include City, State, Country). - Confidence Score: A value between 0 and 1 indicating the model's confidence in the prediction. - Offset: The character position in the original text where the entity begins and ends.

- Categories: The prebuilt NER model supports over 20 entity categories. The most common ones tested on the exam are: - Person: Names of people. - Organization: Companies, institutions, agencies. - Location: Geographic places. - DateTime: Dates, times, durations. - Quantity: Numbers, percentages, currency. - Event: Historical events, holidays. - Product: Products, services. - Skill: Skills or abilities. - Subcategories: Many categories have subcategories. For example, Location includes City, State, Country, Continent, etc. The exam may ask you to identify which subcategory an entity belongs to. - Confidence Score: The model returns a confidence score for each entity. Scores above 0.5 are generally considered reliable, but thresholds can be adjusted in custom implementations. - API Version: The current stable API version for Azure AI Language is 2023-04-01. The NER endpoint is typically https://<your-resource-name>.cognitiveservices.azure.com/language/:analyze-text?api-version=2023-04-01. - Request Format: The API expects a JSON body with a kind field set to EntityRecognition and an analysisInput containing the documents. Each document has an id and text. Example request:

{
  "kind": "EntityRecognition",
  "parameters": {
    "modelVersion": "latest"
  },
  "analysisInput": {
    "documents": [
      {
        "id": "1",
        "text": "Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975."
      }
    ]
  }
}

{
  "documents": [
    {
      "id": "1",
      "entities": [
        {
          "text": "Microsoft",
          "category": "Organization",
          "subcategory": null,
          "offset": 0,
          "length": 9,
          "confidenceScore": 0.99
        },
        {
          "text": "Bill Gates",
          "category": "Person",
          "subcategory": null,
          "offset": 25,
          "length": 10,
          "confidenceScore": 0.98
        },
        {
          "text": "Paul Allen",
          "category": "Person",
          "subcategory": null,
          "offset": 35,
          "length": 10,
          "confidenceScore": 0.98
        },
        {
          "text": "April 4, 1975",
          "category": "DateTime",
          "subcategory": "Date",
          "offset": 50,
          "length": 14,
          "confidenceScore": 1.0
        }
      ],
      "warnings": []
    }
  ],
  "errors": []
}

To use NER, you need an Azure AI Language resource. You can create one via the Azure portal, Azure CLI, or ARM template. The endpoint and key are used to authenticate requests. The exam does not require you to memorize CLI commands, but you should know that you can test NER using:

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://<your-resource-name>.cognitiveservices.azure.com/"
key = "<your-key>"

text_analytics_client = TextAnalyticsClient(endpoint=endpoint, credential=AzureKeyCredential(key))

documents = ["Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975."]
response = text_analytics_client.recognize_entities(documents)

for doc in response:
    for entity in doc.entities:
        print("Entity: {}".format(entity.text))
        print("Category: {}".format(entity.category))
        print("Confidence Score: {}".format(entity.confidence_score))
        print()

NER is often used in conjunction with other Azure AI Language features: - Key Phrase Extraction: Extracts important phrases, which can include entities but is broader. - Sentiment Analysis: Determines sentiment of text, which can be applied to sentences containing entities. - Language Detection: Identifies the language of text, which is necessary for NER to work correctly (the model is language-specific). - PII Detection: A specialized form of NER that identifies personally identifiable information like phone numbers, email addresses, and social security numbers.

On the exam, you may be asked to differentiate between NER and PII detection. NER extracts general entities (Person, Organization), while PII detection specifically targets sensitive personal data. PII detection also redacts the text by replacing entities with their category labels (e.g., "[Person]").

Azure AI Language NER is designed for high throughput. You can send up to 1,000 documents per request (batch). Each document is processed independently. The service scales automatically based on the pricing tier (Free, Standard S). The Free tier allows up to 5,000 text records per month. For production, use the Standard tier with pay-as-you-go pricing.

A large law firm processes thousands of contracts annually. They use Azure AI Language NER to automatically extract parties (Person, Organization), dates, and monetary amounts from each contract. The extracted entities are fed into a database for search and analysis. For example, they can quickly find all contracts where 'Microsoft' is a party or where the contract value exceeds $1 million. The NER model is called via a Python script that reads PDFs, converts them to text, and sends batches of 1,000 documents to the API. The firm uses the Standard tier and processes about 50,000 documents per month. One common issue is that the model sometimes misidentifies a person's name as an organization (e.g., 'Johnson & Johnson' as Organization is correct, but 'Johnson' alone might be tagged as Person). To mitigate this, they use confidence score thresholds—only entities with confidence > 0.8 are accepted. They also run a secondary validation step using a custom NER model trained on legal terminology.

A global e-commerce company receives millions of customer support tickets daily. They use NER to extract product names, locations, and order numbers from ticket text. This information is used to automatically route tickets to the appropriate department. For example, a ticket mentioning 'laptop' and 'New York' is routed to the hardware support team for the Northeast region. The NER API is integrated into their ticketing system via a serverless function that processes each ticket upon creation. The function calls the API with the ticket text and uses the extracted entities to populate routing fields. Performance is critical—they need sub-second response times. Azure AI Language typically responds in under 200ms for short texts. If the API is throttled (due to rate limits), the function retries with exponential backoff. One challenge is that product names often contain numbers (e.g., 'iPhone 14'), which the model may tag as Quantity instead of Product. They address this by using Custom NER to train a model on their product catalog.

A media monitoring company aggregates news articles from thousands of sources. They use NER to extract entities like persons, organizations, and locations to build a knowledge graph of relationships. For each article, they extract entities and then link them to a knowledge base (e.g., Wikidata) using entity linking (another Azure AI Language feature). This enables queries like 'Find all articles where Elon Musk mentions Tesla.' They process millions of articles per day using a distributed pipeline. The pipeline splits large articles into chunks (under 5,120 characters) and sends them in parallel to the API. They use the batch API to maximize throughput. One common error is that the model may miss entities in noisy text (e.g., HTML artifacts). They preprocess text to remove tags and normalize whitespace. They also use language detection to route text to the correct language model. Misconfiguration of the API key or endpoint is a frequent source of errors—they use Azure Key Vault to store secrets securely.