GPT-OSS vs. Qwen vs. Deepseek: Comparing Open Source LLM Architectures

OpenAI’s release of GPT-OSS marks its re-entry into the open-weights model landscape, inviting a closer architectural comparison with other leading models like Alibaba’s Qwen-3 and DeepSeek V3.

OpenAI GPT-OSS

GPT-OSS is a Mixture-of-Experts (MoE) model released in two sizes: 120B and 20B parameters, with each token activating the top four experts. This approach enables efficient inference while leveraging the power of a large parameter count.

• Architecture: It is a decoder-only transformer incorporating modern LLM features:
  • Grouped-Query Attention (GQA): Reduces memory usage and accelerates inference by allowing multiple query heads to share key-value pairs.
  • SwiGLU Activations: Used in feed-forward layers for more nuanced transformations compared to simpler functions like ReLU.
  • Rotary Positional Embeddings (RoPE): Encodes token positions directly within the attention mechanism to better handle long contexts.
  • RMSNorm with Pre-normalization: A normalization technique that enhances training stability.
• Long Context: A standout feature is its native 131,000-token context window. This is achieved by applying Yarn scaling directly during the pre-training phase, rather than as a later fine-tuning or inference-time adjustment.
• Tokenizer & Data: It uses OpenAI’s open-source 200K Harmony tokenizer, a byte-pair encoding (BPE) tokenizer. The model was trained on a text-only corpus of trillions of tokens, with a focus on STEM, coding, and general knowledge.
• Release & Post-Training: The model is released in a quantized format by default, making it accessible for consumer-grade hardware. It underwent significant post-training for safety and alignment, though the open-source community is exploring ways to access the raw model’s capabilities.

Alibaba’s Qwen-3

Released in April, the Qwen-3 family includes both dense and MoE models, with its benchmark scores rivaling other top open-source models.

• Architecture: The models share a core architecture similar to GPT-OSS (GQA, SwiGLU, RoPE, RMSNorm).
  • Variants: It offers dense models from 0.6B to 32B parameters and MoE models with 128 total experts (eight activated per token).
  • Key Innovation: It replaces the static QKV bias of previous versions with QK Norm, which dynamically rescales query and key vectors to maintain stable attention scores at scale.
• Training Process: Qwen-3 was trained on 36 trillion tokens in a three-stage process:
  1. General Stage: Training on over 30 trillion tokens in 119 languages.
  2. Reasoning Stage: An additional 5 trillion high-quality tokens focused on STEM, reasoning, and coding.
  3. Long Context Stage: The context length was extended to over 32,000 tokens using techniques like ABF (Adjusted Base Frequency) for RoPE, Yarn scaling, and Dual Chunk Attention.
• Post-Training Pipeline: Qwen-3 employs a sophisticated four-step post-training process:
  1. Long Chain-of-Thought Cold Start: Fine-tuning on a curated dataset of challenging reasoning problems.
  2. Reasoning RL: Using the GRPO algorithm (developed by DeepSeek researchers) on just 4,000 query-verifier pairs to enhance complex problem-solving.
  3. Thinking Mode Fusion: A key innovation where the model is fine-tuned on a mix of data with and without intermediate reasoning steps, allowing users to toggle a “thinking mode” in the chat interface.
  4. General RL & Distillation: Broadening capabilities in instruction following and tool use, and using strong-to-weak distillation to transfer capabilities from larger to smaller models.

DeepSeek V3 and V3.1

DeepSeek V3 is a massive 671B parameter MoE model designed for both capability and efficiency.

• Architecture: It is an MoE model with several hardware and algorithmic optimizations, including being trained natively in 8-bit precision to reduce costs.
• Attention Mechanism: Unlike GPT-OSS and Qwen-3, DeepSeek V3 uses Multi-head Latent Attention (MLA). MLA compresses keys and values into a smaller latent space before caching, which provides greater memory savings and better modeling performance than GQA, especially for long-context models.
• V3.1 Updates: The latest version builds on the V3 checkpoint, adding a two-phase long-context training approach, a hybrid thinking mode similar to Qwen-3’s, and improved tool use from more advanced post-training.

Comparative Analysis & Reflections

While these models show surprisingly similar top-line benchmark performance, their underlying methods differ significantly.

• Model Sizes: Qwen-3 offers both dense and MoE variants. DeepSeek V3 is a massive 671B MoE model (37B active parameters). GPT-OSS sits in the middle with two MoE sizes (117B with 5.1B active and 21B with 3.6B active).
• Long-Context Strategies: The approaches to extending context length using Yarn highlight different philosophies:
  • GPT-OSS: “Born with it” – Yarn is applied during pre-training for a native 131k context.
  • DeepSeek: “Trained into it” – A staged fine-tuning approach first reaches 32k tokens, then further training achieves 128k.
  • Qwen: “Pushed to the limit” – The model is fine-tuned to 32k, and then at inference time, Yarn scaling is applied again to reach 128k tokens without additional long-context training.

Key Takeaways

The development of these models reveals several trends in the field:

• Empirical Findings: Progress is often driven by empirical discovery. Labs report the combination of techniques that works for them, often without a first-principles justification for why one method (like MLA vs. GQA) is fundamentally superior.
• Convergent Performance from Divergent Methods: Different training recipes, architectures, and techniques are leading to models with very similar benchmark scores, which is a surprising outcome.
• Efficiency of Reinforcement Learning: RL is a critical component of post-training, and it can be remarkably data-efficient, as shown by Qwen-3 achieving strong reasoning with only 4,000 data pairs.
• Data Engineering as a Moat: The specifics of training datasets are often opaque. The immense, behind-the-scenes work in data curation and engineering likely constitutes a significant competitive advantage, making the models difficult to replicate even with open weights.

Ultimately, it is more insightful to look beyond headline benchmarks and context sizes to understand the specific methods and nuanced differences that labs are employing to achieve state-of-the-art results.

• Tags:
• Large Language Models
• Reinforcement Learning
• Gpt-Oss
• Mixture-of-Experts
• Long Context
• Qwen-3
• DeepSeek V3
• Model Architecture