Research notes
Methods, architecture, and evaluation from the lab.
- Project Pigeon: a small model that holds long contextResults from three internal checkpoints. The 865M v7 stays within a few points of the much larger v5 on most shared benchmarks, leads on Winogrande, and passes a long-context retrieval test that both larger checkpoints fail.
- Adversarial robustness in domain-specific models: red-teaming beyond the generic benchmarkGeneric robustness benchmarks miss the failure modes that matter in specialized domains. Robustness for these models should be measured in the learned representations, not the outputs alone.
- Interlocking specialized models: routing and merging domain experts for compound AI systemsA compound system built from a shared pretrained trunk and interchangeable expert heads. The open problems sit in the routing and merging that keep the composition stable.
- Improving synthetic data generation bounds via constrained decodingA decode-time bound on the error of synthetically generated training data. Constraining the generator, rather than filtering its output, keeps a growing corpus on-distribution.