What it is
Nanochat was a foundational experiment that shaped the entire approach to domain-specific AI: is it better to train on massive general data or smaller, highly focused domain data?
The Experiment
Two models, same architecture, different training data:
- MedGen: 500K examples from general PubMed abstracts
- OncoSpec: 10K examples specifically curated for oncology
The Surprising Result
OncoSpec outperformed MedGen on oncology-specific tasks despite having 50x less training data. The lesson was clear: focus beats volume for specialized applications.
Why This Matters
This finding fundamentally changed the approach to OncoVLM and subsequent models. Instead of chasing scale, the focus shifted to data quality and domain relevance. A smaller model trained on the right data beats a larger model trained on everything.
Technical Details
Built using the nanochat framework (Karpathy's nanoGPT lineage) for rapid experimentation. Training runs completed on DGX Spark, with systematic hyperparameter sweeps to ensure fair comparison.
The Framework
Nanochat became more than an experiment: it became the experimental framework for all subsequent model training. The data pipeline, evaluation suite, and training infrastructure built here powered OncoVLM's development.
Legacy
This project validated the hypothesis that personal GPU infrastructure could produce research-grade insights. One person, one DGX, results that challenge conventional wisdom about scale.
Connected work