Multimodal Document Analysis With Ocr

---
name: multimodal-document-analysis-with-ocr
description: "This tool builds a high-precision document parsing pipeline: using **GLM-OCR** for layout element extraction, calling **GLM-4.7** for logical interpretation of table data, and calling **GLM-4.6V** for multimodal visual interpretation of images and charts."
category: "Media"
author: community
version: "1.0.2"
icon: image
---

# GLM-OCR Multimodal Deep Analysis

This tool builds a high-precision document parsing pipeline: using **GLM-OCR** for layout element extraction, calling **GLM-4.7** for logical interpretation of table data, and calling **GLM-4.6V** for multimodal visual interpretation of images and charts.

## Pipeline Implementation Architecture

This Skill consists of two core script stages, orchestrated through `glm_ocr_pipeline.py`:

### 1. Extraction Stage (`scripts/glm_ocr_extract.py`)

- **Core Model**: GLM-OCR
- **Function**: Responsible for physical layout analysis of documents
- **Output**: Extract table HTML and clean to Markdown, automatically crop independent chart image files based on Bbox coordinates, and generate intermediate JSON containing full page reading order

### 2. Understanding Stage (`scripts/glm_understanding.py`)

- **Core Model**: GLM-4.7 (text) / GLM-4.6V (visual)
- **Function**: Responsible for deep semantic reasoning of content
- **Logic**:
  - **Tables**: Combine full text context, use GLM-4.7 to analyze business meaning of Markdown table data
  - **Charts**: Combine full text context + cropped images, use GLM-4.6V for multimodal visual analysis

## Invocation Methods

### Command Line Invocation

```bash
# Run complete pipeline: extraction -> cropping -> understanding analysis, supports input in .pdf, .jpg, .png and other formats
python scripts/glm_ocr_pipeline.py \
  --file_path "/data/report_page.jpg" \
  --output_dir "/data/output"
```

## API Parameter Description

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| file_path | string | ✅ | Absolute path to input file (supports .pdf, .png, .jpg) |
| output_dir | string | ✅ | Result output directory (used to save cropped images and JSON reports) |

## Return Result Structure (JSON)

The tool returns a list containing layout elements and their deep understanding:

```json
[
  {
    "type": "table",
    "bbox": [100, 200, 500, 600],
    "content_info": "| Revenue | Q1 |\n|---|---|\n| 100M | ... |",
    "deep_understanding": "(Generated by GLM-4.7) This table shows Q1 2024 revenue data. Combined with the 'market expansion strategy' mentioned in paragraph 3 of the body text, it can be seen that..."
  },
  {
    "type": "image",
    "bbox": [100, 700, 500, 900],
    "content_info": "/data/output/images/report_page_img_2.png",
    "deep_understanding": "(Generated by GLM-4.6V) This is a system architecture diagram. Visually, it shows the flow of clients connecting to servers through a Load Balancer. Combined with the title 'Fig 3' and context, this diagram is mainly used to illustrate..."
  }
]
```

## Environment Requirements

- Environment variable `ZHIPU_API_KEY` must be configured
- Python 3.8+
- Dependencies: `zhipuai`, `pillow`, `beautifulsoup4`

## Notes

### 1. Model Routing Strategy

- **Table (表格)**: Content passed to **GLM-4.7**, combined with full text Markdown context for logical reasoning
- **Image (图片)**: Image Base64 encoded and passed to **GLM-4.6V**, combined with OCR-extracted titles and full text context for multimodal understanding

### 2. Context Association

All understanding is based on the complete layout logic of the document (Markdown Context), not isolated fragment analysis.

### 3. PDF Processing

Multi-page PDFs default to processing the first page. For batch processing, please extend the loop logic at the script level.