Software Engineer · ML Research · Seattle, WA
Nolan Miller
Software engineer with over 10 years of experience on performance-critical systems. At Google Research I work on the inference efficiency of Transformer-based language models, through fundamental research, system design and hardware co-design. I'm drawn to problems at the intersection of cutting-edge research and large-scale systems.
- Inference efficiency
- Model architecture search
- Efficient attention
- Retrieval-augmented generation
- In-context learning
- Meta-learning
- LLM fine-tuning
Try it live
Apps
Publications
All on Scholar →-
Long context in-context compression by getting to the gist of gisting
Reducing the computational burden of long contexts via compressed representations.
-
Learning and unlearning of fabricated knowledge in language models
How new knowledge propagates through an LLM during gradient updates, and heuristics for keeping updates self-consistent.
-
Narrowing the focus: Learned optimizers for pretrained models
Meta-learning optimizers for more efficient fine-tuning of fixed foundation models.
-
Uncovering mesa-optimization algorithms in transformers
In-context learning analysed as an optimization process inside a simplified transformer.
-
Decentralized Learning With Multi-Headed Distillation
Learning across agents with local data and minimal connectivity between them.
-
Training trajectories, mini-batch losses and the curious role of the learning rate
The dynamics of training trajectories under SGD.
-
Meta-Learning Bidirectional Update Rules
A biologically inspired, meta-learned optimization process for neural networks.
Patents 5
- Meta-Learning Bi-Directional Gradient-Free ANNs — Meta-learned alternatives to gradient descent.
- Metabolic Rate Estimation via SpO2 — Measuring basal metabolic rate directly from SpO2 sensor data, without estimating physical activity.
- Multi-Channel Voice Activity Detection — Using multi-channel audio to improve the sensitivity and accuracy of voice activity detection.
- Providing Secure Digital Images — Images that are readable on screen but illegible in a screenshot, using persistence of vision.
- Object Oriented Image Editing — Semantic image understanding for more natural and faster bulk photo editing.
Projects
Model Design Optimization
A framework for rapidly validating and training novel Transformer architectures, with cross-architecture weight transfer that makes distillation far cheaper and decouples training from serving architecture.
Unified Research Infrastructure
Led the rebuild of LLM training infrastructure for ~40 researchers across 5 teams: 2–4× faster training and experiments that scale to 100× larger models.
Retrieval Augmented Memory ↗
Infrastructure for querying an approximate-nearest-neighbour database (ScaNN) inside a network's training loop, and research on using it for scalable corpus tuning.
Efficient Fine-Tuning
Fine-tuning a whole hierarchy of model sizes by zero-shot transfer of the update computed on a single tuned model.
Multi-Channel Diarization ↗
Novel multi-channel algorithms for speaker segmentation, shipped in the Google Recorder app and Cloud Speech-to-Text.
GoogleGoogle Clips ↗
Hands-free camera using on-device ML for automatic shot selection.
GoogleAIY Vision Kit ↗
DIY-friendly kit with a custom ML accelerator for on-device vision applications.
NASA JPLMarsviewer ↗
Image exploration and rendering software for Mars missions.