Meta Trained Llama 3 405B on 15.6 Trillion Tokens and 30.84 Million GPU Hours, and OpenAI's Human Raters Preferred a 1.3 Billion Parameter Tuned Model to the 175 Billion Parameter GPT-3

Meta’s flagship Llama 3 model has 405 billion parameters, the adjustable numbers inside a neural network, and it learned from 15.6 trillion tokens, which are chunks of text a few letters long. The Llama 3.1 model card puts that one model at 30.84 million GPU hours on Nvidia H100 chips, and the whole family of three sizes at 39.3 million. The paper describing it, dated 23 July 2024, says the 405B model trained on up to 16,000 H100s. (Meta’s paper uses the name Llama 3 for the 3.1 models, and so does this article.)

Meta’s paper splits the job into two stages. Pretraining teaches the network language by predicting the next word at enormous scale. Post-training then tunes it to follow instructions and match human preferences. This article walks through both in order, using the numbers the labs printed in their own papers and model cards, and says where a figure comes from.

The text is filtered before anything is learned

Raw web text is full of duplicates, spam and junk, so every lab cleans it first. OpenAI’s GPT-3 paper (2020) starts from 41 monthly snapshots of Common Crawl, a public archive of the web, covering 2016 to 2019. That was 45 terabytes of compressed text before filtering and 570 gigabytes after, roughly 400 billion tokens.

The filter had two parts. OpenAI trained a classifier to tell its higher-quality reference collections from raw Common Crawl, then favoured the documents the classifier scored higher. It also removed near-duplicate documents, including copies shared between datasets. The authors add that a bug in the filtering missed some overlaps with benchmark test sets, and that retraining was too costly to fix it.

Llama 3’s pipeline is longer. Its paper describes removing websites that carry high volumes of personal information or adult content, deleting repeated lines, counting “dirty words” to catch adult sites that block lists miss, and scoring quality with both fast classifiers and slower RoBERTa-based ones. A language detector sorts documents into 176 languages. Whether copying text at this scale is lawful is being tested in court, as three 2026 rulings show. The final mix, Meta says, is roughly 50 percent general knowledge, 25 percent mathematics and reasoning, 17 percent code and 8 percent multilingual text.

Splitting text into tokens

A network does not read letters or whole words. It reads tokens, pieces from a fixed vocabulary. Sennrich, Haddow and Birch showed in 2015 that breaking rare words into smaller subword units, using a method called byte pair encoding, let a translation system handle words it had never seen.

The 2017 Transformer paper used that method with a shared vocabulary of about 37,000 tokens for English and German. Llama 3 uses 128,000: 100,000 from OpenAI’s open tiktoken tokenizer plus 28,000 added for non-English languages. Meta reports that the larger vocabulary raised English compression from 3.17 to 3.94 characters per token, so the same training compute covers more text.

Pretraining is next-word prediction on a very large scale

The design that made this practical is the Transformer, from a team at Google and the University of Toronto that included Ashish Vaswani and Noam Shazeer. It drops the older step-by-step recurrent design and relies on attention, a mechanism that lets every word in a passage weigh every other word. The paper’s big model reached 28.4 BLEU, a translation score, on English to German after 3.5 days on eight P100 GPUs. Its base model had 65 million parameters and trained for 12 hours.

Pretraining then scaled that recipe. GPT-3 has 175 billion parameters, reads a context window of 2,048 tokens and was trained on 300 billion tokens using V100 GPUs on a Microsoft cluster. The InstructGPT paper puts its pretraining at 3,640 petaflop/s-days, a unit equal to 8.64 x 10^19 floating-point operations.

Llama 3 405B has 126 layers and was pretrained for 3.8 x 10^25 operations, which Meta says is almost 50 times the largest Llama 2. Training began with batches of 4 million tokens and grew to 16 million. In the last phase the context window was stretched in six steps from 8,000 to 128,000 tokens, and a final stage called annealing used higher-quality data.

How many tokens a model of a given size should see

A lab with a fixed compute budget has to split it between a bigger model and more text. DeepMind’s Chinchilla paper of March 2022 trained more than 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens. Its finding was that model size and training tokens should grow together, so that every doubling of one is matched by a doubling of the other.

DeepMind tested this with Chinchilla, a 70 billion parameter model on 1.4 trillion tokens that used the same compute as its 280 billion parameter Gopher, which saw 300 billion tokens. The paper reports Chinchilla beat Gopher and GPT-3 across a wide range of tasks, with 67.5 percent on the MMLU benchmark.

Meta ran its own fit. Extrapolating to 3.8 x 10^25 operations suggested a 402 billion parameter model on 16.55 trillion tokens, and the team settled on 405 billion parameters. By plain division, that is about 38 tokens per parameter for Llama 3 405B, 20 for Chinchilla and under 2 for GPT-3.

What goes wrong across 16,000 chips

Training is one synchronised job, so a single failed GPU can stop all of it. Meta’s paper counts 466 job interruptions in a 54-day stretch of 405B pretraining. Of those, 47 were planned and 419 were not, and about 78 percent of the unplanned ones traced to confirmed or suspected hardware faults. GPU problems alone were 58.7 percent. Only three needed manual intervention, and the team kept effective training time above 90 percent.

The model card lists the energy bill. Training the family produced 11,390 tons of CO2 equivalent by location-based accounting, 8,930 of them for the 405B model. On a market-based basis Meta reports zero, because it matched its electricity use with renewable energy. The wider question of data-centre electricity is covered in this look at the IEA and Berkeley Lab projections.

Post-training turns a text predictor into an assistant

A pretrained model continues text. It does not reliably answer questions. OpenAI’s InstructGPT paper laid out the fix in three steps. Contractors wrote example answers, and the model was fine-tuned on them with ordinary supervised learning. Contractors then ranked several model answers to the same prompt, and a reward model was trained to predict their rankings. Finally the language model was tuned with reinforcement learning, using the reward model’s score as the reward.

The data was small. About 40 contractors worked on the project, and the three datasets held about 13,000, 33,000 and 31,000 prompts. The reward model had 6 billion parameters. The paper puts the compute at 4.9 petaflop/s-days for the supervised step and 60 for the full 175 billion parameter reinforcement run, against 3,640 for GPT-3’s pretraining.

On OpenAI’s own prompts, raters preferred the output of the 1.3 billion parameter InstructGPT to that of the 175 billion parameter GPT-3. The full-size InstructGPT was preferred to GPT-3 85 percent of the time, give or take 3 points. On TruthfulQA it gave truthful and informative answers about twice as often. In closed-domain tasks such as summarising a given text, made-up content fell from 41 to 21 percent, and toxic output was about 25 percent lower when the model was asked to be respectful. The paper also reports an “alignment tax,” small drops on benchmarks such as SQuAD and DROP, which mixing in pretraining updates largely removed.

Llama 3 uses the same idea with a different last step. Meta trained a reward model on human preference data, fine-tuned on examples, picked the best of several sampled answers with the reward model, and then applied Direct Preference Optimization, in six rounds. The paper says Meta tried PPO, the algorithm InstructGPT used, and found DPO needed less compute at large scale. The model card summarises the stage as supervised fine-tuning plus reinforcement learning from human feedback. Its table shows Llama 3.1 405B scoring 85.2 on the MMLU benchmark before post-training and 87.3 after.

Safety tests and the model card

Meta measures two things after safety tuning. Violation rate counts answers that break a safety policy, and false refusal rate counts harmless requests the model wrongly turns down. The paper treats them as a pair, since a model that refuses everything has a perfect violation rate. The red team, which attacks the model on purpose, includes specialists in cybersecurity, adversarial machine learning and responsible AI, plus multilingual experts. Meta also released Llama Guard 3, a separate model that screens inputs and outputs.

The summary document that ships with a release is the model card, a format proposed in 2018 by Margaret Mitchell and colleagues for reporting a model’s intended uses and how it performs across groups. The Llama 3.1 card gives about 15 trillion training tokens, a December 2023 knowledge cutoff, a 128,000-token context, the GPU hours and emissions above, and the uses Meta rules out.

Indian models built on Indian compute

A National e-Governance Division release of 24 February 2026 puts the existing national AI infrastructure at 38,000 GPUs, with 20,000 more to be added. On 6 March 2026 Sarvam AI released two open models and said training ran entirely in India on compute provided under the IndiaAI Mission. Sarvam reports 16 trillion tokens for its 30 billion parameter model, which activates 2.4 billion parameters per token, and 12 trillion for its 105 billion parameter model. Its tokenizer covers all 22 scheduled Indian languages. The programme is covered in the IndiaAI Mission explainer and in the piece on AI in India’s own languages.

Sarvam’s post lists three phases of pretraining, then supervised fine-tuning, then a reinforcement learning stage on an asynchronous GRPO-based system, with rewards that combine verifiable checks and rubric-based scoring across maths, coding and tool use.