What it is
Text Diffusion explores a fundamental alternative to how language models generate text. Instead of predicting one token at a time (autoregressive), diffusion models generate all tokens simultaneously through iterative refinement.
The Research Question
Autoregressive models dominate NLP, but they have limitations: slow generation (one token at a time), difficulty with global coherence, and no ability to revise earlier tokens. Can diffusion approaches, which revolutionized image generation, do better for text?
The Approach
Using SEDD (Score Entropy Discrete Diffusion) and nanodiffusion frameworks, this project systematically compares diffusion vs autoregressive approaches on domain-specific generation tasks. Same data, same compute budget, different paradigms.
Why It Matters for Oncology
Medical text has unique characteristics: structured formats, precise terminology, and strict accuracy requirements. Non-autoregressive generation could enable:
- Faster report generation
- Better handling of structured clinical documents
- Global consistency in long-form outputs
Technical Innovation
The challenge with text diffusion is discreteness: you can't add Gaussian noise to tokens like you can to pixels. SEDD solves this with discrete noise processes designed specifically for categorical data.
Connection to Other Work
Builds on the experimental infrastructure from Nanochat, and findings inform the training approaches used in OncoVLM. The personal DGX makes these long-running experiments feasible.
Status
Active research with promising initial results on constrained generation tasks. Full comparison paper in progress.
Connected work