Paper
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
arXiv:2302.13971 · 0▲ · cs.CL
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
LLaMA is a family of transformer language models (7B to 65B parameters) trained only on publicly available datasets, using architectural changes like pre-normalization with RMSNorm, SwiGLU activations, and rotary positional embeddings. The models are trained on far more tokens than Chinchilla scaling laws recommend (up to 1.4T tokens), trading extra training compute for cheaper inference.
Why it matters
The smaller models are cheap enough to run on a single GPU while matching or beating much larger closed models, which lowers the barrier to running and studying capable LLMs locally instead of via proprietary APIs. Because training used only open data, the recipe is reproducible without licensing inaccessible corpora.
Practical takeaway
You can run a 13B model that competes with GPT-3 (175B) on a single V100 for inference, and the released weights became the base for a large ecosystem of fine-tuned open models. Watch for the tokens-over-parameters tradeoff: overtraining a small model longer pays off at serving time.
Key result
LLaMA-13B outperforms GPT-3 (175B) on most benchmarks despite being roughly 10x smaller, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B on common sense reasoning. Caveat: on MMLU (5-shot), LLaMA-65B scores 63.4 versus 67.5 for Chinchilla and 69.3 for PaLM-540B, which the authors attribute to using only 177GB of books in pretraining.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.