Skip to content
Sarvam's 105-Billion-Parameter Model Computes With 10.3 Billion of Them for Each Token, Using the Mixture-of-Experts Design That Runs From a 1991 Paper to Mixtral and DeepSeek
AI & Technology

Sarvam's 105-Billion-Parameter Model Computes With 10.3 Billion of Them for Each Token, Using the Mixture-of-Experts Design That Runs From a 1991 Paper to Mixtral and DeepSeek

Illustration by tuput

English

Sarvam's open 105B model, trained in India on IndiaAI Mission compute, sends each token to 8 of 128 experts and so activates 10.3 billion parameters at a time. The idea goes back to a 1991 paper by Jacobs, Jordan, Nowlan and Hinton and reached language models in a January 2017 paper by Noam Shazeer and colleagues.

· · 9 min read

Sarvam’s 105B model, published as open weights on 6 March 2026, holds 105 billion parameters (the adjustable numbers a network learns during training), but its Hugging Face model card says only 10.3 billion of them are active for any one token, which is a word or a piece of a word. The saving comes from a design called mixture of experts. Sarvam says it trained the model and its smaller 30B sibling from scratch in India, on compute provided under the IndiaAI Mission.

Mixture of experts is also what sits inside Mixtral, DeepSeek-V2 and DeepSeek’s newer models. The papers and model cards trace a line from a 1991 paper to a 2017 network with 137 billion parameters, and from there to the open models of 2024 to 2026. Sarvam’s 105B shares two design features with DeepSeek-V2, a shared expert and a compressed form of attention, both described below.

What a router does inside the network

For readers new to the terms, our explainer on how a large language model is trained covers parameters, tokens and pretraining.

In an ordinary transformer, the design behind today’s chatbots, every token passes through a feed-forward block at every layer. That block is a large stack of numbers that does much of the model’s processing. A mixture-of-experts model swaps the one block for several smaller ones, called experts, and adds a router. Hugging Face’s 2023 explainer describes the router as a small learned network that decides which tokens go to which expert, and says it is trained together with the rest of the model. Mistral’s post on Mixtral says the same: experts and routers are trained at the same time.

The result is a split between two numbers. In Hugging Face’s February 2026 wording, a model’s capacity depends on its total parameters, but its inference speed depends on its active parameters. The post works through the open gpt-oss-20b model: 21 billion parameters in all, 4 of 32 experts used per token, about 3.6 billion active. A speed estimate from an Apple M3 Ultra’s memory bandwidth came to about 111 tokens a second, and the measured speed was about 115.

An expert is not a topic specialist. The 2026 post says it is simply a learnable sub-network, not a “math expert” or a “code expert”. The 2023 post reports that in one study (ST-MoE) experts in the encoder specialised in groups of tokens such as punctuation or proper nouns, and that in a multilingual run no single expert specialised in a single language.

A 1991 idea, a 2017 jump

The starting point is a paper in the journal Neural Computation, volume 3, from 1991, by Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton. It describes a learning procedure for a system of many separate networks, each handling a subset of the training cases, with a gating network that sends each case to one or a few experts so that weight changes stay local to them. A footnote in the paper says the idea was first presented by Jacobs and Hinton at a summer school in Pittsburgh in 1988. The authors tested it by splitting a vowel-recognition task into subtasks, each solved by a very simple expert network.

Noam Shazeer and six co-authors posted the next large step on 23 January 2017. Their abstract reports “greater than 1000x improvements in model capacity” with only minor losses in computational efficiency on modern GPU clusters. The layer they built held up to thousands of feed-forward experts, and a trainable gating network picked a sparse combination of them for each example. They applied one with up to 137 billion parameters between stacked LSTM layers (an older kind of language network) and report better results than the best published at lower computational cost on language modelling and translation benchmarks. Those results are the authors’ own. Hugging Face’s account adds that the work ran into high communication costs and unstable training.

Two simplifications: GShard and Switch

A June 2020 paper on GShard reports scaling a multilingual translation model with sparsely gated experts beyond 600 billion parameters, trained on 2,048 TPU v3 accelerators in 4 days for translation from 100 languages into English.

Switch Transformers, by William Fedus, Barret Zoph and Shazeer, followed on 11 January 2021. Its main change was to simplify routing. Hugging Face notes that earlier designs used at least two experts per token, and that Switch sends each token to one, which cuts router work, communication and batch sizes while, the authors say, preserving quality. The paper reports up to 7 times faster pre-training than its T5-Base and T5-Large starting points on the same hardware, and a trillion-parameter model with a 4 times speedup over T5-XXL. The authors also say they trained large sparse models in lower-precision bfloat16 numbers for the first time.

2024: Mixtral and DeepSeek change the recipe

Mistral AI released Mixtral 8x7B on 11 December 2023 under the Apache 2.0 licence, and its paper followed on 8 January 2024. Each layer holds 8 experts and the router chooses 2 per token. Because the experts differ from token to token, the paper says each token has access to 47 billion parameters but uses 13 billion in a given step. Mistral’s own announcement gives 46.7 billion and 12.9 billion, which round to the same figures, and Hugging Face compares the speed to a 12 billion parameter model. Mistral says Mixtral matches or outperforms Llama 2 70B and GPT-3.5 on most benchmarks, a claim from the maker and measured on its chosen tests.

The total is 47 billion and not 8 times 7 billion, which is 56 billion, because only the feed-forward blocks are split into experts. Attention and the other parts are shared, Hugging Face explains.

Three days after the Mixtral paper, the DeepSeekMoE paper appeared, on 11 January 2024. It made two changes. It cut experts into finer pieces and activated more of them per token, and it set aside some shared experts to hold common knowledge so the routed ones could specialise. In the paper’s tests a 16 billion parameter version matched LLaMA2 7B using about 40% of the computation, and a preliminary 145 billion parameter version matched DeepSeek 67B using 28.5% of it, maybe as little as 18.2%. These comparisons are the authors’ own.

DeepSeek-V2 followed on 7 May 2024: 236 billion parameters in total, 21 billion activated per token, a 128,000-token context window and 8.1 trillion training tokens. Against the earlier dense DeepSeek 67B, the team reports 42.5% lower training cost, a 93.3% smaller key-value cache and up to 5.76 times the generation throughput. The key-value cache is the memory a model keeps of earlier tokens while it writes, and the reduction comes from the second feature in the paper, Multi-head Latent Attention (MLA), which compresses that cache into a smaller stored form.

Sarvam’s two models, built in India

Sarvam’s blog of 6 March 2026 and the two model cards give the design in detail. Both models use sparse expert layers with 128 experts, with one shared expert always on in addition to the routed ones.

ModelTotal parametersActive per tokenRouted experts used
Mixtral 8x7B (paper)47 billion13 billion2 of 8
DeepSeek-V2 (paper)236 billion21 billionDeepSeekMoE design
Sarvam 30B (model card)about 30 billion2.4 billion, non-embedding6 of 128
Sarvam 105B (model card)about 105 billion10.3 billion8 of 128

The card counts Sarvam 30B’s active parameters without the embedding layer, the lookup table that turns tokens into numbers. The Hugging Face file listings show 32 billion parameters for the 30B and 106 billion for the 105B, so the model names are rounded.

The two differ in how they handle attention. The 30B uses Grouped Query Attention, in which several attention heads share stored keys and values (the card lists 4 key-value heads), to shrink the cache. The 105B uses Multi-head Latent Attention, the technique DeepSeek-V2 uses to compress it, which Sarvam describes as further cutting memory for long inputs. Pre-training used 16 trillion tokens for the 30B and 12 trillion for the 105B.

On routing, Sarvam says it scored experts with sigmoid functions instead of the usual softmax and added an expert-bias term. The company’s stated reason is better balance across experts and less routing collapse, a failure in which a few experts take most of the traffic. Hugging Face’s 2023 piece describes load balancing as one of the hard problems of the design, which Switch Transformers handled with a simplified extra loss added during training.

Sarvam released both under Apache 2.0, with weights on AI Kosh and Hugging Face. Its blog says the 30B powers its Samvaad conversation platform and the 105B its Indus assistant. It also reports that on H100 chips the 30B delivers 3 to 6 times the throughput per GPU of a Qwen3 baseline at matching speeds per user. That figure and the company’s claim of state-of-the-art results on Indian-language benchmarks are Sarvam’s own statements. Outside scores for the 105B are covered in our report on the IndiaAI Mission, and the mission’s other release of a sparse model, ZenteiQ’s Dhir 5B, is in this article.

How far the idea has gone

DeepSeek-V4.1-Flash, whose model card was live on the Hugging Face Hub on 12 November 2026, has 552 billion backbone parameters and context of up to 1 million tokens. The card says each sparse layer holds 1 shared expert and 384 routed ones, with 6 routed experts active per token. That is 6 of 384 routed experts, about 1.6%, where Mixtral uses a quarter of its experts and Sarvam’s 105B uses 8 of 128. The card lists a 45 trillion token training corpus and an MIT licence. The same approach at 1 trillion parameters is covered in our article on Mistral Large 4.

What the design does not save

Every expert has to sit in memory even though only a few work on each token. The 32 billion and 106 billion parameter file sizes listed for Sarvam’s models are what a machine must hold. Hugging Face’s 2023 explainer says a Mixtral-class model needs enough video memory to hold the full 47 billion parameters, and that mixture-of-experts suits high-throughput serving across many machines while a dense model is better with little memory and low traffic. It also reports that sparse models have been prone to overfitting when fine-tuned, and that the gains from adding experts shrink after 256 or 512 of them.

Share
Copied!

Sources & further reading

  1. Jacobs, Jordan, Nowlan and Hinton: Adaptive Mixtures of Local Experts, Neural Computation 3, 79-87 (1991)
  2. arXiv: Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (23 January 2017)
  3. arXiv: Lepikhin et al., GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (30 June 2020)
  4. arXiv: Fedus, Zoph and Shazeer, Switch Transformers (11 January 2021)
  5. Mistral AI: Mixtral of experts (11 December 2023)
  6. arXiv: Jiang et al., Mixtral of Experts (8 January 2024)
  7. arXiv: Dai et al., DeepSeekMoE (11 January 2024)
  8. arXiv: DeepSeek-AI, DeepSeek-V2 (7 May 2024)
  9. Hugging Face: Mixture of Experts Explained (11 December 2023)
  10. Hugging Face: Mixture of Experts (MoEs) in Transformers (26 February 2026)
  11. Sarvam AI: Open-sourcing Sarvam 30B and 105B (6 March 2026)
  12. Hugging Face: Sarvam-105B model card
  13. Hugging Face: Sarvam-30B model card
  14. Hugging Face: DeepSeek-V4.1-Flash model card

Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.

#mixture of experts#sarvam ai#sarvam 105b#mixtral#deepseek#switch transformers#open-weight models#indiaai mission

Enjoyed this? Get the next one.

One good read at a time, straight to your inbox. No spam, unsubscribe anytime.

More in AI & Technology
DeepSeek's V4.1-Flash, an Open-Weight 552-Billion-Parameter Model Released on 10 September 2026, Cuts Its Per-Token Memory Cache to 890 Bytes, About a Quarter of Its Predecessor's
A model's short-term memory of a long conversation is expensive to keep. DeepSeek's September release compresses it by design, and its own report says where the model still trails. Here is the mechanism, the timeline from 2024, and what outsiders have checked.
Sarvam Vision 2.1 Reads English and the 22 Scheduled Indian Languages, Adds Handwriting and Form Extraction, and Scored 87.3 on olmOCR-Bench in Sarvam's Own Tests
A Bengaluru company's model now reads forms, tables and handwriting in 22 Indian languages. The release post, the public benchmark pages and the script rules behind the problem, with each score traced to who ran the test.
OpenAI's 2020 Scaling Laws Paper Found Model Loss Falls as a Power Law Over More Than Seven Orders of Magnitude, and DeepMind's 2022 Chinchilla Paper Showed a 70 Billion Parameter Model Could Beat a 280 Billion One on the Same Compute
A model's size, its training text and its compute bill can be traded against each other. Here is what each paper measured, the numbers each printed, one replication attempt, and what a 2026 safety report says about how far the rules reach.
← all articles