Building Better Language Models Through Global Understanding

Dr. Mazi Fadai discusses the critical challenges in multilingual AI, including data imbalances and flawed evaluation methodologies. She argues that tackling these difficult multilingual problems is not only essential for global accessibility but also a catalyst for fundamental AI innovation, much like how machine translation research led to the Transformer architecture. The talk introduces new, more culturally aware evaluation benchmarks like Global MMLU and INCLUDE as a path toward building more robust and globally representative language models.

July 28, 2025 4 min read

Building Better Language Models Through Global Understanding — source: https://www.youtube.com/watch?v=fcStqSMuoYw

The development of truly intelligent systems is fundamentally tied to embracing the world’s linguistic diversity. While modern language models excel in English, this represents only a fraction of human knowledge. Working on multilingual problems is crucial for two primary reasons:

Accessibility and Equity: It ensures that the benefits of AI technology are accessible to people globally, not just speakers of high-resource languages.
Driving Innovation: Tackling the complex, multifaceted challenges of multilingualism forces the development of more sophisticated and generalizable models.

Historical Perspective: Multilingualism as a Catalyst for Innovation

The history of Natural Language Processing (NLP) shows that breakthroughs often emerge from the challenges of multilingual tasks. The journey to the current state-of-the-art generative models began in the field of machine translation (MT):

Until 2014, Statistical Machine Translation (SMT) was the dominant paradigm, using n-gram language models primarily to improve the fluency of translations.
In 2014, a series of papers introduced Recurrent Neural Networks (RNNs) for MT, first as a scoring component and then as a full end-to-end generative model.
The introduction of the attention mechanism (first global, then local) allowed models to selectively focus on relevant parts of the input sentence, overcoming the bottleneck of compressing the entire input into a single vector.
Innovations like Byte-Pair Encoding (BPE) enabled more effective tokenization for morphologically rich languages and handling of rare words.
This progression culminated in the Transformer architecture in 2017, a model born from the need to solve the machine translation task more effectively, which has since become the foundation for most modern large language models (LLMs). This history demonstrates that solving hard multilingual problems leads to powerful, general-purpose solutions.

The Crisis in Modern LLM Evaluation

A significant obstacle to progress is the current state of evaluation, which often lacks the rigor needed for meaningful comparison, especially in a multilingual context. A robust evaluation framework should be:

Fair: How can we fairly compare models with different architectures, training data, decoding methods, and language coverages?
Transparent: Are evaluation conditions, such as prompts, hyperparameters, and potential data leakage, clearly reported?
Representative: Do benchmarks measure what truly matters to users across different cultures? Do they contain culturally relevant questions and account for cultural nuances?
Robust: Are results replicable and statistically significant? How much can we trust aggregate scores?

Recent analysis reveals several inconsistencies in common evaluation practices:

Translation Quality of Benchmarks: The performance of a model can change significantly depending on which translation service (e.g., Google Translate, NLLB) was used to create the multilingual version of an evaluation set. This introduces a major confounding variable.
The Danger of Aggregation: Averaging scores across all languages can mask critical weaknesses. A model that ranks highly on average might perform poorly on medium or low-resource languages. For instance, a model could be #1 on the MMLU benchmark but drop to #4 when evaluated only on low-resource languages. Users need detailed, per-language results to choose the right model for their specific use case.
Statistical Significance: Many multilingual benchmarks have very small test sets for individual languages (e.g., 100 samples or fewer). Performance differences on such small samples are often not statistically significant, making it difficult to draw reliable conclusions.
LLM-as-a-Judge: Using an LLM to judge the output of another is a common practice for open-ended generation. However, there is no single “best” judge; different judge models (e.g., GPT-4, Command R) show varying levels of correlation with human preferences across different languages.

A Path Forward: Building Better, Culturally-Aware Benchmarks

To guide progress effectively, the community must invest in creating better and more representative evaluation benchmarks.

Global MMLU: An analysis of the popular MMLU benchmark revealed that 28% of its questions are culturally sensitive, with a staggering 86% of those being Western-centric. Simply translating these questions is insufficient, as the underlying cultural context does not transfer. To address this, the Global MMLU project was created. It involved:
- Translating MMLU into over 40 languages.
- Having human annotators label each question as “culturally sensitive” or “culturally agnostic.”
- Performing post-editing to correct translation errors and ensure cultural appropriateness. This allows for a more nuanced evaluation, revealing how models perform differently on universal knowledge versus culturally specific knowledge.
INCLUDE Benchmark: To complement translated benchmarks, the INCLUDE project focuses on sourcing original, in-language questions. Instead of translating from English, this initiative collects multi-choice questions from native sources like university entrance exams in 44 languages. This captures local knowledge and linguistic styles that are impossible to obtain through translation.
Multimodal Benchmarks: This effort has been extended to multimodal contexts. The Aya Vision Bench provides a benchmark for evaluating image understanding and open-ended generation in a multilingual setting, recognizing that visual context is often intertwined with cultural and linguistic understanding.

Conclusion

To build models that possess true intelligence and global understanding, we must move beyond an English-centric view. This requires a disciplined and scientific approach to evaluation, emphasizing transparency, per-language reporting, and statistical significance. Ultimately, progress depends on global collaboration—including people from diverse linguistic and cultural backgrounds in the creation of data, benchmarks, and models. This is not just about making technology accessible; it is about building fundamentally better and more capable AI.