Expert parallelism at hero scale
Expert Parallelism made a 535B-A23B model practical to train. We compare FSDP, fixed-capacity EP, and results from a scaling ladder.
From the open frontier
Marin Blog
Research, experiments, and updates from the Marin community. Read original Marin posts alongside selected writing from Open Athena.
Curating 25 Trillion Tokens for LLM Pretraining
How Marin deduplicates and decontaminates permissively licensed data, scores quality, tags topics, and learns how often to sample each data bucket.
Marin 535B-A23B launch note
A look back at Marin's first year at Open Athena and forward at the 535B-A23B hero run, the largest model the team has ever trained.
Marin's 535 billion parameter model training run launched with support of The Jen-Hsun and Lori Huang Foundation GPU gift
Open Athena announces the launch of Marin's largest training run yet—a 535B parameter large language model—with a generous gift of compute from The Jen-Hsun and Lori Huang Foundation.
A 1B standard Transformer rivals Evo 2 40B on variant effect prediction
How MarinDNA combined data curation, hyperparameter transfer, scaling, and data-mixture experiments to build an efficient 1B alignment-free genomic language model with balanced performance across genomic regions.
Open Development of Frontier AI
Open development is key to making frontier-level artificial intelligence legible, reusable, and public.
Cluster Scheduling with Iris
Training a frontier-level LLM requires significant resources across a variety of providers and accelerator types. Our Iris scheduling system has allowed us to effectively make use of these precious resources using a custom global scheduler; in the months since rollout, our sustained concurrent TPU usage has roughly doubled.
Improving our LLM Pretraining Efficiency
How Marin pretraining became more efficient through Mixture of Experts, higher expert sparsity, MuonH, PKO, and routed expert normalization.
Async RL from Scratch on TPUs
Scaling Laws That Extrapolate 300× Past the Fit
Delphi is an open scaling suite ranging from 3e18 to 1e23 FLOPs. A pre-registered forecast from its scaling law predicted the loss of the largest run within 0.2%, extrapolating 300× past the largest run used in the fit.
Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scale
Quantile Balancing (QB) is a hyperparameter-free load balancer for Mixture of Experts models, introduced by Jianlin Su. We validated it on a 32B-A5B (1e22 FLOPs) Marin run over 326B tokens: zero hyperparameters, zero loss spikes, and no need for leading dense layers, auxiliary losses, or capacity overload factors.
Marin 32B Retrospective
Scaling the Marin recipe to 32B: loss spikes, recovery, QK-Norm, midtraining, and lessons learned.
Marin 8B Retrospective
The first Marin 8B run: its changing data mix, training phases, mistakes, and lessons learned.