OncoVLM: Domain-Specific Foundation Models
Proving that focused training beats raw scale
Training oncology-specific multimodal models that outperform larger general-purpose models. Multi-teacher knowledge distillation at three scales: 4B, 1.7B, and 500M parameters.
The problem
The Scale Assumption
The AI field assumed bigger models were always better. But for specialized domains like oncology, general-purpose 70B models often missed domain-specific nuances that smaller, focused models could capture.
- General models lacking oncology-specific knowledge
- Expensive inference costs for large models
- No multimodal understanding of pathology/radiology
- Hallucinations in clinical contexts
What I built
Multi-Teacher Distillation
Instead of training one massive model, we distilled knowledge from multiple specialized teachers into smaller, focused students optimized for oncology tasks.
- Three model scales: 4B, 1.7B, 500M parameters
- Multi-teacher distillation from MedGemma, GPT-OSS-20B, Qwen-3-30B
- Multimodal: pathology images, radiology, clinical text
- LoRA fine-tuning for parameter efficiency
How it fits together
Architecture
The training pipeline uses DGX Spark's 128GB VRAM for full-batch training with automated experiment tracking.
Timeline
How it went
- Mar 2025Nanochat experiments validate focused training
- Apr 2025Initial OncoVLM architecture design
- May 20254B model training complete
- Jun 20251.7B and 500M variants trained
- Jul 202592.4% PubMedQA achieved
What it taught me
Lessons
- 10K focused examples can outperform 500K general examples
- Multi-teacher distillation captures complementary strengths
- Personal GPU infrastructure enables research-grade experiments
- Smaller models can beat larger ones on specialized tasks
Stack
Connected work