Concept
Knowledge distillation
Knowledge distillation is a training technique where a small "student" model learns to imitate a larger "teacher" model, rather than learning only from the raw labels. The idea, popularized by Hinton et al.
The rest of “Knowledge distillation” is a premium feature: every concept in the library gets a precise, practitioner-focused write-up like this one, cross-linked straight from the paper summaries that use it.
Log in to unlock→