Fully Connected Tokyo: [Hands-on workshop] Automation of document workflows in financial industry

Upstage’s core vision is to transform unstructured documents into actionable business outcomes, a concept termed “Document to Decision”. The primary challenge in this process, especially within the financial industry, is converting vast amounts of non-standardized information—from scanned papers to complex PDFs—into a clean, structured, “AI-ready” format that Large Language Models (LLMs) can effectively utilize.

The Two-Stage Pipeline: Document AI and LLM

The proposed solution is a two-stage pipeline that assigns distinct, specialized roles to different AI models for optimal performance and accuracy.

1. Stage 1: Upstage Document Parse (Document AI) The first step utilizes Document Parse, a sophisticated AI model that goes beyond traditional Optical Character Recognition (OCR). Its main purpose is to analyze the visual layout of a document and convert it into a structured, machine-readable format like HTML. This process correctly interprets complex elements such as multi-column layouts and tables, preserving the relationships between different pieces of information. This structural understanding is critical for preventing the context-related errors and “hallucinations” that can occur when an LLM processes poorly parsed text.

2. Stage 2: Large Language Model (LLM) for High-Level Tasks Once the document is converted into clean, structured text, it is passed to an LLM (such as Upstage’s “Solar” model). With the properly formatted input, the LLM can reliably perform a variety of high-level tasks based on user prompts, including:

   • Information Extraction: Pulling specific data points from the document.
   • Question Answering: Responding to queries based on the document’s content.
   • Summarization & Classification: Condensing information or categorizing documents for routing and analysis.

Hands-On: Building a Robust Extraction Pipeline

The workshop demonstrated a practical scenario of processing insurance accident reports, which often come in inconsistent templates with potentially missing information.

A simple, single-prompt approach might extract the necessary information, but it often fails to meet enterprise requirements for consistency, formatting, and system integration. A more robust method involves a multi-step prompting strategy:

• Prompt 1: Define Scope: Instruct the model to focus only on extracting specific, predefined items.
• Prompt 2: Apply Formatting Rules: Provide explicit instructions for data normalization. This includes rules for handling missing information (e.g., outputting “記載なし” or “not available”), standardizing date and time formats (e.g., converting Japanese calendar years to Gregorian, using a 24-hour clock), and ensuring consistent terminology.
• Prompt 3: Enforce Output Schema: Mandate that the final output be in a machine-readable format like JSON with a predefined structure and keys. This ensures the data can be seamlessly ingested by downstream business systems.

This pipelined approach transforms unstructured documents into clean, reliable, and system-ready structured data. The process was tracked and visualized using the Weights & Biases Weave tool, which provides observability into each step of the AI application.

Systematic Evaluation for Production Readiness

Building the pipeline is only half the process; ensuring its accuracy and reliability is paramount. The workshop introduced a systematic evaluation framework using W&B Weave.

1. Create a “Golden” Dataset: A set of test documents is curated with corresponding “ground truth” labels—the exact, correctly formatted data that the pipeline is expected to produce.
2. Define Scorers: Custom functions, or “scorers,” are created to automatically compare the model’s output against the ground truth. The workshop demonstrated two types:
   • Exact Match Scorer: For data like dates or ID numbers, this scorer checks for a perfect string match.
   • Embedding Similarity Scorer: For free-text fields (e.g., accident descriptions), where phrasing can vary, this scorer uses vector embeddings to measure semantic similarity rather than relying on an exact match.
3. Analyze and Iterate: An evaluation dashboard provides a detailed, field-by-field breakdown of the pipeline’s performance. This allows developers to quickly identify patterns in errors (e.g., consistently failing to format a driver’s license number correctly) and provides clear, actionable insights for refining prompts or models.

Real-World Business Impact in Insurance

Upstage shared two key case studies from the financial sector:

• Hanwha Life Insurance: The company had 5 million unstructured medical claim documents from the past three years that were effectively unusable. By processing 240,000 documents per day with Document Parse, they were able to structure this data for the first time. This enabled large-scale data analysis that led to the development of a new, highly profitable insurance product in just four months.
• AIA Life Insurance: To manage the high cost of processing small claims, the company implemented an AI-powered automation system. By combining Document AI and an LLM with business rules (e.g., claims below a certain value are fully automated), they increased their automated payment rate from 35% to 60% and boosted the rate of payments made within one business day from 65% to 83%. This demonstrates how AI can balance accuracy, cost, and customer satisfaction.

For sensitive data, Upstage offers multiple deployment options, including on-premise and private cloud installations, to meet the security and compliance needs of financial institutions.

• Tags:
• LLM
• Evaluation
• RAG
• Feature
• W&B Weave
• Insurance
• Finance
• Document AI
• Information Extraction
• OCR