Stories about LoRA-MoE
1 related stories
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
AI InsightCoRe-MoE proposes a compact reusable MoE for continual multimodal instruction tuning. It finds that LoRA update direction subspaces overlap across tasks, so new tasks can be expressed via lightweight coordinates instead of full experts, significantly reducing parameter overhead compared with prior LoRA-MoE. This shifts continual learning from expanding experts to sharing subspaces.Key TakeawayInstead of training full LoRA experts per task, CoRe-MoE reuses shared direction subspaces via lightweight coordinates.Why It MattersContinual instruction tuning often suffers from parameter growth and catastrophic forgetting; shared subspaces may support more tasks with fewer parameters, improving efficiency.Who's Affected- AI ResearchersProvides a reusable-subspace continual learning method that can reduce storage and compute for later tasks.
- DevelopersCould enable more economical continual expansion of multimodal model capabilities with lower fine-tuning costs.
What's NextWatch for forgetting control and capacity limits over long task sequences, and reproducibility at larger scale.Importance 70/100