PII Detection and Extraction

Personally Identifiable Information (PII) refers to any data that can be used to identify an individual. Examples include names, addresses, phone numbers, email addresses, social security numbers, passport numbers, credit card numbers, and medical record numbers. PII detection is the process of automatically identifying such information within text documents. This is crucial for compliance with data protection regulations like GDPR, HIPAA, and CCPA, which require organizations to protect sensitive data. Azure AI Language provides a pre-built PII detection feature that can identify, categorize, and optionally redact PII entities from text.

Azure AI Language uses a combination of machine learning models and rule-based pattern matching to detect PII. The service is part of the Azure AI Language (formerly Text Analytics) API, which is a cloud-based service that provides natural language processing capabilities. When you submit a text document, the service performs the following steps:

Azure AI Language supports a comprehensive list of PII entity types. These are organized into categories. On the AI-900 exam, you are expected to know the most common ones:

For a complete list, refer to the Azure documentation. The exam may ask you to identify which entity type is detected from a given example.

To use PII detection, you call the REST API endpoint: https://<your-resource-name>.cognitiveservices.azure.com/language/:analyze-text?api-version=2023-04-01. The request body includes the text and the kind of analysis: PiiEntityRecognition. Here is an example request:

{
  "kind": "PiiEntityRecognition",
  "parameters": {
    "modelVersion": "latest",
    "domain": "phi",
    "piiCategories": ["USSocialSecurityNumber", "CreditCardNumber"]
  },
  "analysisInput": {
    "documents": [
      {
        "id": "1",
        "language": "en",
        "text": "My SSN is 123-45-6789 and my credit card is 4111-1111-1111-1111."
      }
    ]
  }
}

The domain parameter can be set to "phi" (Protected Health Information) to include healthcare-specific entities. The piiCategories parameter allows you to filter which entity types to detect. If omitted, all supported types are detected.

The response includes a list of detected entities with their offsets, lengths, types, and confidence scores:

{
  "kind": "PiiEntityRecognitionResults",
  "results": {
    "documents": [
      {
        "id": "1",
        "redactedText": "My SSN is *********** and my credit card is ****************.",
        "entities": [
          {
            "text": "123-45-6789",
            "category": "USSocialSecurityNumber",
            "offset": 9,
            "length": 11,
            "confidenceScore": 0.85
          },
          {
            "text": "4111-1111-1111-1111",
            "category": "CreditCardNumber",
            "offset": 42,
            "length": 19,
            "confidenceScore": 0.95
          }
        ],
        "warnings": []
      }
    ],
    "errors": [],
    "modelVersion": "2023-01-01"
  }
}

Note the redactedText field: it shows the original text with detected entities replaced by asterisks of the same length. This is the default redaction behavior. You can also choose to redact with a tag by setting the redactionPolicy parameter to "maskWithEntityType".

PII detection is often used in conjunction with other Azure AI services:

The PII detection API supports up to 125,000 characters per document and up to 1,000 documents per request (batch). The service has a throughput limit based on your pricing tier (Free tier: 5,000 transactions per month, Standard tier: varies). For high-volume scenarios, consider using asynchronous batch processing with the /analyze endpoint (long-running operations).

You can test PII detection using: - Azure Language Studio: A no-code UI where you can paste text and see detected entities interactively. - REST API: Use tools like Postman or curl. - SDKs: Available for Python, C#, Java, JavaScript, and Go.

Example curl command:

curl -X POST "https://<your-resource-name>.cognitiveservices.azure.com/language/:analyze-text?api-version=2023-04-01" \
-H "Ocp-Apim-Subscription-Key: <your-key>" \
-H "Content-Type: application/json" \
-d '{
  "kind": "PiiEntityRecognition",
  "parameters": {
    "modelVersion": "latest"
  },
  "analysisInput": {
    "documents": [
      {
        "id": "1",
        "language": "en",
        "text": "Contact me at john.doe@example.com or call (555) 123-4567."
      }
    ]
  }
}'

This understanding is critical for the AI-900 exam, where you may be asked about the capabilities, limitations, and appropriate use cases of PII detection.

A hospital system needs to de-identify patient notes before sharing them with researchers. They use Azure AI Language's PII detection with the domain parameter set to 'phi' to detect Protected Health Information (PHI) such as medical record numbers, health plan IDs, and patient names. They process thousands of documents daily via a batch pipeline using Azure Data Factory. The redacted notes are stored in Azure Blob Storage with access restricted to authorized researchers. A common misconfiguration is forgetting to set the domain parameter, which would miss PHI-specific entities like 'HealthPlanID' or 'MedicalRecordNumber'. Performance considerations: the Standard tier handles up to 1,000 documents per batch, and they monitor API usage to avoid throttling. When the API returns low confidence scores for certain entities, they implement a manual review process using Azure Machine Learning to improve model accuracy.

A financial services company wants to automatically redact sensitive information from customer support emails before storing them in a CRM. They use Azure Logic Apps to trigger PII detection whenever a new email arrives. The email body is sent to the API, and the redacted text is saved to the CRM. The company configures the piiCategories parameter to only detect credit card numbers, social security numbers, and bank account numbers to avoid over-redacting names (which are needed for customer identification). A common pitfall is that the API may detect false positives for credit card numbers in strings of digits that are not valid credit card numbers (e.g., a long order number). The company mitigates this by using a confidence threshold of 0.9 and by implementing a post-processing step that validates the Luhn algorithm for credit card numbers.

A legal firm uses Azure Cognitive Search to index internal documents. Before indexing, they run PII detection to redact sensitive client information. They use the redaction policy 'maskWithEntityType' to replace each PII entity with its category name (e.g., '[SSN]') for easier review. The firm processes documents in batches of 500 using the asynchronous batch API to handle large volumes without hitting rate limits. A common issue is that the API may not detect all PII due to unusual formatting (e.g., SSN written as '123456789' without dashes). To address this, they preprocess documents to normalize formats before detection. They also use custom entity recognition for domain-specific identifiers not covered by the built-in types.