Document Layout Analysis

Document Layout Analysis is a computer vision technique that identifies and classifies the physical structure of a document page. Unlike Optical Character Recognition (OCR), which extracts raw text and its bounding boxes, layout analysis groups text and images into meaningful regions: paragraphs, sections, tables, figures, headers, footers, page numbers, and selection marks (checkboxes and radio buttons). This is a prerequisite for downstream tasks like document understanding, form extraction, and intelligent search.

Azure AI Document Intelligence (formerly Form Recognizer) offers a prebuilt layout model that performs this analysis. It is part of the broader Document Intelligence service, which includes prebuilt models for invoices, receipts, identity documents, and custom extraction models. The layout model is the foundation: you can run it alone to understand a document's structure, or use its output as input to other models.

The layout model uses a deep neural network based on a transformer architecture, specifically a variant of the LayoutLM family. The process has four stages:

3. Layout Classification: The model analyzes the spatial arrangement of text blocks, images, and lines. It uses attention mechanisms to understand relationships between adjacent elements. For each detected region, it assigns a type from a predefined taxonomy: - paragraph: A block of text that forms a coherent paragraph. - sectionHeading: A heading that starts a new section. - figure: An image or chart (bounding box only, no content extraction). - table: A structured table with rows and columns. - pageNumber: A page number. - header/footer: Repeating text at the top or bottom of the page. - checkbox/radioButton: Selection marks (checked or unchecked). - formula: A mathematical equation.

4. Output Generation: The model returns a JSON object containing: - pages: Array of pages, each with width, height, angle, and units (pixels or inches). - paragraphs: Array of paragraphs with content, boundingRegions (page, polygon), and spans (offset and length in the concatenated text). - tables: Array of tables with cells (each with kind (content, rowHeader, columnHeader), rowIndex, columnIndex, rowSpan, columnSpan, content, boundingRegions). - figures: Array of figures with boundingRegions and caption (if detected). - sections: Array of section headings with content and boundingRegions. - selectionMarks: Array of selection marks with state (selected or unselected), boundingRegions, and confidence.

To call the layout model, use the Azure AI Document Intelligence REST API or SDK. Here is an example using the Python SDK:

from azure.ai.formrecognizer import DocumentAnalysisClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://your-endpoint.cognitiveservices.azure.com/"
credential = AzureKeyCredential("your-api-key")
client = DocumentAnalysisClient(endpoint, credential)

with open("document.pdf", "rb") as f:
    poller = client.begin_analyze_document("prebuilt-layout", document=f)
    result = poller.result()

for page in result.pages:
    print(f"Page {page.page_number}: width={page.width}, height={page.height}, angle={page.angle}")

for table in result.tables:
    print(f"Table with {len(table.cells)} cells")
    for cell in table.cells:
        print(f"  Row {cell.row_index}, Col {cell.column_index}: {cell.content}")

for mark in result.selection_marks:
    print(f"Selection mark at page {mark.page_number}: state={mark.state}, confidence={mark.confidence}")

To verify the output, examine the JSON response. The analyzeResult object contains all layout elements. For example, a table cell appears as:

{
  "kind": "content",
  "rowIndex": 0,
  "columnIndex": 0,
  "rowSpan": 1,
  "columnSpan": 1,
  "content": "Name",
  "boundingRegions": [{"pageNumber": 1, "polygon": [x1,y1,x2,y2,x3,y3,x4,y4]}]
}

Document Layout Analysis is often the first step in a pipeline:

A large logistics company receives 50,000 invoices per month in PDF format. They need to extract line items, totals, and vendor names. Before extraction, they use the layout model to identify tables (line items) and paragraphs (vendor address). The layout output is fed into a custom extraction model that uses the table structure to extract each row. The company processes invoices in batches using Azure Logic Apps, which calls the layout API and stores the JSON in Azure Blob Storage. A key challenge is handling multi-page invoices where tables span pages; the layout model correctly identifies the table as a single entity with cells on multiple pages. Misconfiguration: If the company uses the read model instead, they lose the table structure and must parse text manually, leading to 20% error rates.

A law firm uses Document Intelligence to analyze contracts. They need to identify clauses (paragraphs), signatures (figures), and checkboxes (selection marks). The layout model helps them categorize each section. They run the layout model on thousands of documents and then use custom NLP to extract clause types. Performance: Processing a 100-page contract takes about 2 minutes. The firm must handle scanned documents with low resolution; they preprocess images to 300 DPI to improve accuracy. A common problem is that handwritten signatures are often misclassified as figures, which is acceptable because they only need the bounding box for redaction.

A hospital digitizes patient intake forms. The forms contain checkbox fields (e.g., symptoms list) and handwritten notes. The layout model correctly identifies checkboxes and their state (selected/unselected). However, handwritten text in note fields is not extracted by the layout model; they use a separate handwriting recognition model. The hospital processes 10,000 forms daily using a batch pipeline. They set a confidence threshold of 0.7 for selection marks to reduce false positives. Misconfiguration: If they do not filter by confidence, they might mark a smudge as a selected checkbox, causing incorrect patient records.