Introducing Our Approach to Design Document Review Using Business-Specific Large Language Models

Hitachi’s Digital Systems & Services Sector is pursuing a 30% increase in system development productivity by leveraging Generative AI. This initiative stems from the need to address Japan’s declining labor population. This presentation focuses on a specific application within the Financial Business Unit, which develops mission-critical systems like core banking and ATMs, where quality and security are paramount.

The Challenge: Limitations of General-Purpose LLMs

The Financial Business Unit’s development process involves highly specialized terminology, procedures, and extensive design documentation. Initial attempts to use general-purpose LLMs and Retrieval-Augmented Generation (RAG) proved insufficient. These methods struggled to achieve the required accuracy for practical business use due to a lack of deep understanding of the specific domain.

Solution: A Domain-Specific LLM for Design Document Review

To overcome these limitations, the team developed a specialized LLM focused on reviewing design documents. The core of their approach was to enhance an open-source LLM through a two-step process:

1. Continued Pre-training (APT - Adaptive Pre-training): The model was pre-trained on a vast corpus of internal data, including:
   • Past design documents from multiple projects (tens of gigabytes).
   • Quality guidelines and checklists.
   • Textual data and diagrams/figures (図表), with diagrams converted to Mermaid format to make their structure understandable to the LLM.
2. Instruction Fine-Tuning (SFT - Supervised Fine-tuning): The model was then fine-tuned to perform specific review tasks.

Initially, the team experimented with full fine-tuning but found it time-consuming and inefficient. They pivoted to LoRA (Low-Rank Adaptation), a Parameter-Efficient Fine-Tuning (PEFT) method, which allowed for faster iteration and more efficient experimentation. The final architecture evolved into a multi-agent system, where a general-purpose LLM acts as an orchestrator, delegating specialized queries to the domain-specific LLM they developed. This hybrid approach proved more effective than relying on a single monolithic model.

MLOps and Infrastructure

The development was managed using the Weights & Biases (W&B) platform, specifically their dedicated cloud BYOB (Bring Your Own Backend) service. This setup was crucial for security, as it allowed Hitachi to keep sensitive project data on their on-premise servers while using W&B’s cloud environment for managing metadata, experiment tracking (via Workspaces, Artifacts, etc.), and collaboration. This significantly improved productivity and traceability compared to using disparate OSS tools.

Training was conducted on Hitachi’s own high-performance GPU servers, which required significant infrastructure upgrades, including new power supplies and advanced cooling systems to handle the immense heat generated.

Evaluation and Tangible Results

The team built an automated evaluation pipeline using an LLM-as-a-judge approach to compare models and score them against human-prepared evaluation datasets. This automation was critical for rapid, iterative development.

The results were significant:

• Workload Reduction: The tool automated the tedious process of cross-referencing multi-hundred-page design documents against checklists with over 100 items, achieving a 70% reduction in manual workload.
• Accuracy and Usability: Approximately 50% of the LLM’s output is directly usable, while the remaining 50% serves as a high-quality draft that requires human refinement. This drastically reduces the initial cognitive load on engineers.
• Superior Performance over RAG: In a concrete example, the fine-tuned LLM was able to locate and verify specific details about performance testing in a document, whereas a RAG-based approach failed to even identify the relevant information.

Future Outlook

The specialized LLM has been integrated into Hitachi’s standard development framework. The long-term vision is to move beyond simply identifying defects in documents to creating a full-cycle AI agent that can also propose and implement corrections. The goal is to incrementally automate the entire document creation, review, and revision process to enhance both quality and efficiency in large-scale system development.

• Tags:
• LLM
• Fine-Tuning
• Feature
• Enterprise AI
• MLOps
• Multi-Agent Systems
• LoRA
• Weights & Biases
• Domain-Specific LLM
• System Development
• Quality Assurance