Zai GLM 4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

The GLM-4.6 model series represents a significant milestone in the development of open-source models, achieving performance on par with leading commercial models like GPT-4o and Claude 3.5 Sonnet on the LMSYS Chatbot Arena. This success is built on a comprehensive and evolving technical roadmap, starting from the first GLM-130B model in 2022 to a family of over 65 models that have surpassed 100 million downloads.

The Training Recipe: A Multi-Stage Approach

The performance of GLM-4.6 stems from a meticulously structured, five-stage training pipeline designed to build a strong foundation and then progressively instill specialized reasoning and agentic capabilities.

1. General Pre-training: The process begins with a 15 trillion token dataset of general-purpose data, including web pages, books, and multilingual content. This stage builds a strong, all-around base model with a context length of 4,000 tokens.
2. Reasoning Continued Training: On top of the base model, an additional 7 trillion tokens of code and reasoning data are introduced. This dataset is composed of high-quality open-source repositories, as well as math, science, and logic problems that feature step-by-step reasoning.
3. Repo-Level Code Training: To enhance coding abilities, the training shifts to “repo-level” data. Instead of isolated code snippets, the model is fed entire contexts, including multiple files, issues, and pull requests from the same project. This teaches the model to understand cross-file dependencies and real-world project structures. At this stage, the context length is expanded to 32,000 tokens, allowing the model to process the key files of a medium-sized repository in a single pass.
4. Synthetic Reasoning Data: Approximately 500 billion tokens of synthetic reasoning data are added. This data covers math, science, and algorithms with explicit “thinking traces,” laying the groundwork for future agent behaviors like task decomposition, reflection on mistakes, and long-chain reasoning.
5. Long Context and Agent Data: Finally, the model is trained on 100 billion tokens of long-context and agent-specific data. The context length is pushed to 128,000 tokens (and up to 200,000 for GLM-4.6), enabling it to handle entire documents or codebases. This data includes a high volume of agent trajectories, such as multi-step tool calls, search queries, and code execution logs, to explicitly train its capabilities as an AI agent.

SLIME: An Efficient Reinforcement Learning Framework

To effectively train agentic behaviors, Z.ai developed SLIME, a hybrid reinforcement learning (RL) framework built on an asynchronous inference stack. It is designed to handle the diverse computational needs of different RL tasks.

• Synchronous Mode: For short reasoning tasks like math or code completion, SLIME operates in a synchronous mode where training and inference occur on the same GPU. This allows the next batch of data to be sampled immediately from the latest policy, maximizing GPU utilization and learning efficiency.
• Asynchronous (Decoupled) Mode: For complex agent tasks that involve interacting with external environments (e.g., calling a backend API, browsing a website), a synchronous approach would create bottlenecks. In this mode, SLIME decouples the system:
  • An inference cluster interacts with the agent environments, generates trajectories, and writes them to a shared data buffer.
  • A training cluster consumes this data at its own pace to update the model and periodically pushes the new model weights to the inference cluster. This design ensures that slow environmental interactions do not stall the entire training pipeline.

SLIME also incorporates efficiency optimizations, such as using BF16 for the main training loop while running data generation and inference with FP8 quantization for higher throughput without sacrificing training stability.

Key Lessons from Reinforcement Learning

Through experimentation, the team identified several key principles for effective RL training:

• Single-Stage RL for Long Context: When fine-tuning a model that already has long-context capabilities (e.g., 64,000 tokens from SFT), a multi-stage RL approach that starts with shorter contexts (e.g., 16k -> 32k -> 64k) is detrimental. It causes the model to “forget” its long-context abilities. The better approach is to start RL training directly at the full 64,000-token context length in a single stage.
• Two-Stage Difficulty Curriculum: Instead of a fixed dataset, a curriculum learning strategy is more effective. The model is first trained on medium-difficulty problems to ensure meaningful reward gradients. Once it becomes stronger, it is challenged with extremely hard problems in a second stage, allowing it to continue improving.
• Token-Weighted Loss for Code: For coding tasks, averaging the loss over individual tokens rather than the entire sequence leads to faster, more stable convergence. This method also reduces the model’s tendency to generate short, templated answers just to secure a reward.
• Data Quality Over Quantity: In scientific reasoning tasks (tested on GPQA), training on a small, clean dataset of expert-verified questions yielded significantly better performance than training on a much larger, mixed-quality dataset. This highlights that for complex reasoning, data quality is more critical than raw data size.

GLM-4.5V: Native Resolution and Video Understanding

The GLM family also includes the powerful multimodal model, GLM-4.5V. Its architecture consists of a Vision Transformer (ViT) encoder, an MLP projector, and the GLM-4.5 language model base.

A key feature is its ability to process images and videos at their native resolution and aspect ratio, avoiding the information loss that occurs when inputs are forced into a fixed square size. This is particularly crucial for understanding UI screenshots, documents, and slides. For video, the model ingests temporal information by inserting time-index tokens after each frame, helping it understand temporal order and action sequences. These capabilities also power its function as a GUI agent, where it can control a computer or browser by interpreting the screen.

Deployment and Ecosystem

GLM-4.6 and GLM-4.5V are available as open-source weights and are integrated with popular inference frameworks like vLLM and SGLang from day one. For users without access to large-scale GPU resources, the models are accessible via the Z.ai platform and its corresponding API.

• Tags:
• AI Agents
• Reinforcement Learning
• Multimodal Models
• Open-Source AI
• LLM Training
• GLM-4