What it is
OncoVLM represents a systematic approach to building domain-specific multimodal AI for oncology. The project started with the hypothesis that smaller, focused models could outperform larger general-purpose models on specialized tasks.
The Multi-Scale Architecture
- 4B parameter flagship model for comprehensive analysis
- 1.7B parameter model balancing capability and efficiency
- 500M parameter model for edge deployment and rapid inference
Training Methodology
Used multi-teacher knowledge distillation, combining insights from MedGemma (medical domain expertise), GPT-OSS-20B (general reasoning), and Qwen-3-30B (multilingual capabilities). This approach allowed the smaller models to inherit specialized knowledge from multiple larger teachers.
Key Results
- 92.4% accuracy on PubMedQA benchmark
- Outperforms general-purpose models of similar size on oncology tasks
- Successfully handles multimodal inputs: pathology images, radiology, and clinical text
Technical Stack
Built on Gemma architecture with PaliGemma for vision encoding. Fine-tuned using LoRA for parameter efficiency, enabling training on DGX Spark's 128GB VRAM. The distillation pipeline runs automated experiments using the Autonomous Researcher framework.
Connected work