Writing, building, thinking — out loud.

Making it yours: distillation and fine-tuning a domain model

# Summary This post examines how fine-tuning and "distillation" are often conflated in modern LLM training, despite being fundamentally different mechanisms. Fine-tuning is simply continued training on small datasets that adjusts model weights, while true distillation (as originally defined) involves training a smaller student model to match a teacher's output probability distributions. The author argues that many training tools mislabel fine-tuning as "distillation," obscuring the distinction between these two approaches and what they actually accomplish.