GShard (600B)
No estimate
No hardware requirements for this model
The weights for this model have not been published, so it cannot be downloaded or run on your own hardware at any size. It is reachable only through its provider, and no graphics card changes that.
On record
Full specification
Everything on record for this model. Most of it describes how it was trained rather than how it runs — useful context for judging how much work went into it, and how it compares with models built at a different scale.
Origin
Who built this model, where, and when it was published.
- Organisation
- Organisation type
- Industry
- Country
- United States of America
- Published
- 30 June 2020
- Authors
- Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen
What it does
The problem areas the model was built for. A model can carry several of each.
- Domain
- Language
- Task
- Translation
- Approach
- Self-supervised learning
Size
How large the model is and how much data it was trained on. Parameters are the figure that decides whether it fits on a given graphics card.
- Parameters
- 600B
- Training data
- 1,000,000,000,000 tokens
"The 600B parameters model that achieved the best translation quality was trained with 2048 TPU v3 cores for 4 days, a total cost of 22 TPU v3 core-years."
"We focus on improving the translation quality (measured in terms of BLEU score [48]) from all 100 languages to English. This resulted in approximately 13 billion training examples to be used for model training" Each example is a sentence pair. Assuming 20 words per sentence and 4/3 tokens per word, that is 13*20*4/3 billion tokens
Training compute
The arithmetic performed to train the model, measured in floating-point operations. It is a measure of what the training run cost, not of how fast the finished model answers you.
- Training compute
- 1.3 × 10²² FLOP
- How it was established
- Third-party estimation,Hardware
https://arxiv.org/ftp/arxiv/papers/2104/2104.10350.pdf Table 4 123000000000000 FLOP / chip / sec * 96360 chip-hours [see training time notes] * 3600 sec / hour * 0.3 [assumed utilization] = 1.2800462e+22 FLOP
The training run
What it physically took to train: which chips, how many, for how long, and what that drew from the wall.
- Training hardware
- Google TPU v3
- Chip-hours
- 96,360
- Wall-clock time
- 96 hours
https://arxiv.org/pdf/2104.10350 Table 4 gives 3.1 days "The 600B parameters model that achieved the best translation quality was trained with 2048 TPU v3 cores for 4 days, a total cost of 22 TPU v3 core-years." 22 TPU v3 core-years * 365 days per year * 24 hours per day / 2 cores per TPU v3 chip = 96360 chip-hours
Availability
Whether you can obtain the model and run it on your own hardware, which is what decides if any of the graphics-card figures on this page apply.
- Weights
- Closed — provider access only
- Model access
- Unreleased
- Training code
- Open source
training code is open, Apache: https://github.com/tensorflow/lingvo/tree/master/lingvo/tasks/lm
How it is classified
Labels the source dataset applies when tracking notable models, and how confident it is in the entry.
- Record confidence
- Confident
- Citations
- 1,992
Sources
Where this record came from and when it was last checked.
- Reference
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Last updated
- 25 May 2026
What the numbers mean
About this model
GShard (600B) was published by Google, in United States of America, in June 2020. It comes out of industry.
It works in Language, and is recorded as doing translation.
Its weights were never published, so it can only be reached through its provider. No graphics card changes that.
How it was trained
Training it took roughly 1.3 × 10²² FLOP of computation, on Google TPU v3 — a measure of what producing the model cost, not of how fast it answers.
Around 1,000,000,000,000 tokens went into training it.
Answers
GShard (600B) — common questions
When was GShard (600B) released?
GShard (600B) was published in June 2020. Capability per parameter has improved considerably since, so a newer model of the same size is often the better use of the same hardware.
What is GShard (600B) used for?
GShard (600B) works in Language, and is recorded as handling translation. These are the areas it was designed around; they describe intent rather than a hard boundary.
How much compute was used to train GShard (600B)?
Around 1.3 × 10²² FLOP, on Google TPU v3. That measures what producing the model cost and says nothing about how quickly it answers once trained — inference speed comes from memory bandwidth, not from the training budget.
What GPU do I need to run GShard (600B)?
None. GShard (600B) is a closed model — its weights were never published, so it cannot be downloaded or run on your own hardware at any price. It is reachable only through its provider.
Is GShard (600B) open source?
No. GShard (600B) has not had its weights published, so it exists only as a service controlled by its owner.
How many parameters does GShard (600B) have?
GShard (600B) has 600B parameters. "The 600B parameters model that achieved the best translation quality was trained with 2048 TPU v3 cores for 4 days, a total cost of 22 TPU v3 core-years.". That figure is the total, and it is what decides how much memory the model needs — roughly half a gigabyte per billion at the compression most people use.
Who created GShard (600B)?
GShard (600B) was published by Google, based in United States of America, categorised as industry.
The other direction
Looking at it from the other side?
This page starts from the model. If you already own a card and want to know everything it will run, start from the hardware instead.