How Kimi Was Built
Zhilin Yang stood on a stage at GTC and explained, in remarkable detail, exactly what his team did. Here is what he said — in plain English.
A note before we begin: this article follows a technical keynote delivered by Moonshot AI co-founder Zhilin Yang at NVIDIA's GTC 2026 conference. Yang was speaking to an audience of AI researchers and engineers. We are simplifying deliberately, and where we simplify, we say so. The ideas here are his. The plain English is ours.
By Aaron Rose · Tech Reader Magazine · July 19, 2026
Building An AI That Keeps Getting Better
Zhilin Yang walked onto the GTC stage with a single animating question: how do you build an AI model that keeps getting better? Not by throwing more raw computing power at the problem in the same old ways. His answer was more interesting than that. He had three specific ideas — three distinct levers his team pulled — and the model they produced by pulling all three at once is Kimi K2.5.
Yang is a co-founder of Moonshot AI, the Beijing-based lab behind the Kimi family of models. His talk was dense, fast, and aimed squarely at people who already spend their days training neural networks. What follows is that talk, unpacked for everyone else.
The Starting Point: Why "More" Isn't Enough
Yang opened with a slide that most people in AI consider settled truth: if you train a model on more data, using more computing power, you get a smarter model. This relationship is called a scaling law, and it has been the engine of AI progress for years. More tokens — the chunks of text a model learns from — means lower loss, which roughly means better performance.
But Yang's point was that simply doing more of the same has a ceiling. The world only contains so much high-quality text to learn from. If you've already trained on 50 trillion tokens of the best available data, you can't just conjure 50 trillion more. So his team asked a different question: what if you could get more intelligence out of the same data?
That question led them to three separate areas of work, each attacking the problem from a different angle. Yang called them his three scaling dimensions.
The world only contains so much high-quality text.His team asked a different question:what if you could get more intelligence out of the same data?
Dimension One: A Better Engine Under the Hood
The first dimension is about how a model actually learns during training. When an AI model trains, it adjusts billions of internal settings — called weights — based on feedback about how wrong its predictions were. The process that manages those adjustments is called an optimizer. For years, the default optimizer used by almost everyone in the field has been a method called Adam, invented in 2014.
Yang's team switched to something different: an optimizer called Muon. The core idea behind Muon is that when it makes an adjustment to the model's weights, it ensures each adjustment is mathematically independent from the others — pointing in a different direction, so that each nudge is doing unique work rather than overlapping with what other nudges are already doing.
This sounds abstract, but the practical result is striking. Using Muon instead of Adam, with the exact same model and the exact same training data, produced roughly twice the token efficiency. In Yang's framing: if you had 50 trillion tokens of high-quality data and your optimizer was twice as efficient, it's as if you magically had 100 trillion tokens. The data didn't change. The model didn't get bigger. The optimizer got smarter about learning from what was there.
2×Approximate token efficiency improvement when replacing the Adam optimizer with Muon — using the same model, the same data, and the same compute.
Scaling Muon to very large models turned out to be harder than it looked. When Yang's team tried to train a one-trillion-parameter model with Muon, they ran into instability: a number called the max logit — essentially a measure of how confident the model is — kept exploding to values far outside normal ranges, and the training eventually fell apart.
Their fix was a technique they call QK-Clip. During training, the system monitors that runaway number in real time and applies a gentle constraint to keep it in a safe range. Crucially, the fix doesn't hurt the model's learning — Yang showed training curves that were identical before and after applying it, except that the instability was gone. The Kimi K2.5 training run went through more than 15 trillion tokens without a single loss spike. Yang called it one of the most beautiful training curves he'd ever seen.
Dimension Two: Handling More Pages
The second dimension is about how much a model can hold in its head at once. This is called context length — essentially, how many words the model can read and reason about in a single sitting before it starts forgetting what it saw earlier.
Yang made an interesting historical observation here. Before transformers — the architecture that underlies virtually every modern AI model — the dominant approach used a different type of network called an LSTM. LSTMs were good at short sequences but hit a wall. Add more context and their performance stopped improving. Transformers don't have that wall. The more context you give them, the better they get, and that improvement keeps going as context grows.
This matters enormously for anything that requires sustained reasoning over long documents, large codebases, or extended tasks. An AI that can only hold a few pages in mind at once is fundamentally limited in what it can do. Kimi K2.5 was built to hold vastly more.
To get there efficiently, Yang's team built a new architecture they call Kimi-Linear. The key innovation is in how it handles attention — the mechanism by which a model decides which earlier parts of its context are relevant to what it's doing right now. Standard full attention is powerful but expensive: the computational cost grows rapidly as context gets longer. Kimi-Linear mixes a more efficient form of attention (called linear attention) with standard full attention in a roughly one-to-three ratio, getting most of the benefit at a fraction of the cost.
The linear attention component was itself redesigned. Think of it like human memory: you remember your childhood home address for decades, but you forget what you had for lunch last Tuesday within days. Different kinds of information decay at different rates, and that's not a bug — it's useful. The original linear attention design used a single global setting for how fast the model should forget, which meant it had to pick one rate for everything. Yang's team replaced that single dial with many independent ones, one per information channel. Some channels hold onto things across a very long range. Others refresh quickly to absorb new input. The result is a memory system that can decide, for each kind of information it's tracking, how long to hold on.
Some channels hold information across a very long range.Others refresh quickly.The model can decide, for different kinds of information, how long to hold on.
Dimension Three: A Team, Not a Single Mind
The third dimension is the most conceptually different from the first two. It doesn't improve the model itself — it changes how the model is deployed to solve hard problems.
Yang's team calls this agent swarms. The idea is simple to describe: instead of sending a single AI agent to work on a complex task, you send many. An orchestrating agent — think of it as the manager — breaks a large task into subtasks and assigns them to a group of specialized sub-agents working in parallel. Those sub-agents return their results, the orchestrator assembles them, and the final answer is richer and faster than any single agent could have produced alone.
Yang used a company analogy that maps cleanly. A CEO doesn't personally write every report, build every feature, or verify every fact. The CEO decomposes the goal, assigns the work, and synthesizes what comes back. That's exactly what the orchestrating agent does.
Training this system required new reward signals — ways of telling the model what good behavior looks like. Three were used. The first rewarded the model for actually spawning sub-agents rather than defaulting to doing everything itself (a collapse the team called "serial collapse"). The second rewarded the model for making sure the tasks it spawned were actually completed, not just created and abandoned. The third rewarded the overall outcome — did the final result actually answer the question? All three signals were applied together, with their weights adjusted over the course of training as the model got better at each.
The Result? Kimi K2.5
Putting all three dimensions together — Muon for token efficiency, Kimi-Linear for long context, agent swarms for complex task execution — produced Kimi K2.5, released by Moonshot AI in early 2026.
One aspect of K2.5 that Yang spent time on was the way it handles vision. Most AI models that can see images learned to do so after the fact — text training happened first, vision was added later as a kind of retrofit. Kimi K2.5 was trained differently. Vision and text were fused from the very first token of training, a approach the team calls early fusion.
The result surprised even Yang's team. The two modalities actively helped each other rather than trading off. Training the model on purely visual tasks — counting objects, answering questions about images, with no math or coding involved — improved its performance on text-only reasoning tasks. Vision made the model better at thinking, even when there was nothing to look at.
The reverse was equally striking. A model with a strong text foundation turned out to need almost no dedicated visual training data at all. Yang called this zero vision SFT: the team used essentially no vision-specific supervised training, relying instead on the strength of the shared representation the two modalities had built together. The model still performed at near state-of-the-art on visual benchmarks.
Some of the capabilities that emerged from this joint training were not ones the team explicitly designed for. A model that can read a video and generate a working website styled after that video — replicating its visual identity in code — is doing something that requires vision and language and reasoning to be woven together at a deep level, not bolted together after the fact. That's what early fusion produced.
A Glimpse at What's Next
Yang closed with a preview of a new architectural idea his team published the day before the talk. It doesn't have a simple name yet — the paper calls it attention residue — but the concept connects neatly to everything that came before.
Residual connections are one of the foundational ideas of modern deep learning. Before them, stacking layer after layer into a deep network caused training to collapse — the signal that guides learning would either vanish or explode before it could reach the early layers. Residual connections fixed this with a simple trick: each layer adds its output to whatever came in, rather than replacing it. That change made deep networks trainable, and it has been standard ever since.
Yang observed that a residual connection is, in a certain mathematical sense, a recurrent network rotated ninety degrees — operating through the depth of the network rather than through time. And if that's true, then just as you can upgrade a simple recurrent network to a more powerful attention mechanism operating through time, you can do the same thing through depth. Instead of each layer just adding to the previous layer's output, it attends to all previous layers' outputs simultaneously.
Early results show a 24% improvement in token efficiency from this architectural change alone — on top of the gains from Muon and Kimi-Linear. Yang presented it not as a finished product but as the direction his team is moving. The next generation of Kimi models, he suggested, will be built on it.
24%Additional token efficiency gain from attention residue —the next-generation architectural change Yang previewed at the close of his talk.
Three scaling dimensions. A new optimizer. A new architecture. A new way of deploying models as coordinated teams. A training approach that fuses vision and language from day one. And now, a new way of thinking about the connections between layers themselves. Yang didn't present any of this as finished. He presented it as a direction — one his team has been following consistently, publishing as they go, with Kimi K2.5 as the current proof that the direction is working.
The Source Material
Zhilin Yang's full GTC 2026 keynote — "How We Scaled Kimi K2.5" — is available on YouTube via the Kimi AI channel. The technical papers behind Muon, Kimi-Linear, and attention residue are available on arXiv.
Tech Reader Magazine
TechReaderMagazine.com