You guys — I just went down a rabbit hole with Hugging Face's latest post about moving past LoRA, and my brain is still buzzing! Everyone talks about LoRA like it's magic, but here's what they forget: it's actually only low-rank updates to the weights. That means even if you give it good data, there's a hard ceiling on how much knowledge a rank-constrained adapter can absorb. But then we have QLoRA — which has already become one of the most popular alternatives because its quantization is way cleaner than any other method out there, and the memory savings for production workloads are genuinely huge compared to full finetuning.
But here's what really blew me away: Dora. It uses weight decomposition instead of low-rank matrices and cuts training parameters by about 40% while capturing much more complex knowledge than a LoRA ever could. The team laid out this fantastic comparison — for an LLM in the 7B range, full finetuning is actually feasible with enough compute, but if you're limited, QLoRa at 4-bit quantization wins on memory every time compared to vanilla methods. They even show that Dora outperforms standard LoRA on multiple benchmarks when your model needs deep specialization from a smaller dataset.
The thing I keep coming back to — and this is what makes the article worth bookmarking forever - is their "how to choose" decision tree at the end because they don't declare one winner, which is so rare in these things! Small dataset with low compute? Go LoRA for simplicity and 16-bit precision. Need high-quality data that needs deep learning from a small sample size? Use Dora for the decomposition bonus. Producing on large models where memory matters? QLoRA at 4 bits beats every other adapter method in production. And don't discount full finetuning — when you have the budget and capacity, going all-in still holds value over adapters because it avoids rank caps entirely.
Source: https://huggingface.co/blog/peft-beyond-lora
Also see: hftaromeo/model_cards_with_metadata
But here's what really blew me away: Dora. It uses weight decomposition instead of low-rank matrices and cuts training parameters by about 40% while capturing much more complex knowledge than a LoRA ever could. The team laid out this fantastic comparison — for an LLM in the 7B range, full finetuning is actually feasible with enough compute, but if you're limited, QLoRa at 4-bit quantization wins on memory every time compared to vanilla methods. They even show that Dora outperforms standard LoRA on multiple benchmarks when your model needs deep specialization from a smaller dataset.
The thing I keep coming back to — and this is what makes the article worth bookmarking forever - is their "how to choose" decision tree at the end because they don't declare one winner, which is so rare in these things! Small dataset with low compute? Go LoRA for simplicity and 16-bit precision. Need high-quality data that needs deep learning from a small sample size? Use Dora for the decomposition bonus. Producing on large models where memory matters? QLoRA at 4 bits beats every other adapter method in production. And don't discount full finetuning — when you have the budget and capacity, going all-in still holds value over adapters because it avoids rank caps entirely.
Source: https://huggingface.co/blog/peft-beyond-lora
Also see: hftaromeo/model_cards_with_metadata