Research Notes

14 November 2026

• Distributing full-matrix EMA-Gram eigensolves across TPUs

  Engineering · A 924-update Delphi comparison led us to adopt whole-matrix eigensolve sharding, with faster training updates on a larger TPU mesh and mixed early validation loss.

• Avoiding GPU memory spikes when switching MoE layouts

  Engineering · An evaluation handoff left communication buffers live on the GPUs. Preserving the placement of the weight shards removed that allocation in the tested layout.

• Mochi’s 262K loss gap persisted after removing retrieval evidence

  Modeling · Log scaling lowered Mochi’s loss on a 262K retrieval test. Most of the gap remained when we removed the earlier replies from the prompt and scored the same target reply.

• Factoring Gated DeltaNet-2’s decay for TPU matrix multiplies

  Engineering · A September Gated DeltaNet-2 prototype ran in about one-third of the Atomic Ops kernel time in single-TPU tests. We explain how it reuses decay factors across token pairs.

• Fewer tokens and longer time to target with one-eighth-power EMA-Gram

  Modeling · In a single-seed 447M experiment, one-eighth-power EMA-Gram reached the C4 loss target with 2.9% fewer tokens than zero power and took 7.47× as much elapsed time on the TPU setups tested.