Let's Learn About Knowledge Distillation!

Knowledge Distillation is exceedingly easy to do and has been around since the inception of large models. Since it cannot be performed by a 5th grader, it remains a complete black box to most. All of a sudden, people with money do not like Knowledge Distillation. So, in order to look like they are smarter than a 5th grader, everyone all of a sudden is talking about Knowledge Distillation. The people with money who build the models also do not like getting sued. They have utilized one singular argument since the inception of this in every lawsuit, they are not actually touching or storing the data directly itself. I agree with every frontier model provider that has ever made this argument. They are correct. It is exactly why they win their lawsuits. Knowledge Distillation falls into literally the same category. Every single argument that the frontier model providers utilize, have utilized, and will continue to utilize in defense of this, is also applicable to Knowledge Distillation. You cannot just carve it out. Cake for me but not for thee? So, what exactly is it that people are asking for when they make these arguments? Do you like getting sued? Because making these arguments as a frontier model provider, is how you lose lawsuits. It is the most short sighted argument you could ever make. I Can't Read I Only Like Video