What is KD (Knowledge Distilation) that DeepSeek allegedly used?

Chinese AI startup DeepSeek has sent ripples through the AI community with the unveiling of its large language model, Deepseek-R1. This model boasts performance on par with leading models like OpenAI's GPT, yet reportedly requires significantly less cost and training time. One of the key techniques DeepSeek purportedly leveraged is 'Distillation.' Let's delve into the mechanism, advantages, and disadvantages of this approach.

Knowledge Distillation (KD) is a technique where a smaller, simpler model (the "Student") learns to mimic the behavior of a larger, more complex model (the "Teacher"). The "Teacher" model, trained on a vast amount of data, possesses a rich understanding of the underlying patterns and relationships within that data. The "Student" model, with its reduced size and complexity, is then trained to replicate the performance of the "Teacher," effectively inheriting its knowledge.

The origins of this concept can be traced back to the 2006 paper "Model Compression." Caruana et al. used a large ensemble model, consisting of hundreds of base classifiers (which was the state-of-the-art at the time), to label a massive dataset. They then trained a single neural network on this newly labeled dataset using traditional supervised learning. This compressed model, despite being a thousand times smaller and faster than the original ensemble, achieved comparable performance.

Since then, KD has been successfully applied in various domains, including Natural Language Processing (NLP), speech recognition, image recognition, and object detection. Recently, it has gained prominence in large language model (LLM) research, emerging as an effective method for transferring the advanced capabilities of leading proprietary models to smaller, more accessible open-source models.

Types of Knowledge Distillation

• Response-based Distillation: This is the most common type, where the Student learns to mimic the final predictions or output probabilities of the Teacher.
• Feature-based Distillation: In this approach, the Student is trained to replicate the intermediate representations or feature maps generated by the Teacher model. These feature maps capture the internal representation of the data learned by the Teacher model at various layers of the network. By mimicking these representations, the Student can gain a deeper understanding of the data and potentially learn more robust features.
• Relation-based Distillation: This focuses on transferring the relationships between different layers or data samples learned by the Teacher. For example, the Student can learn to capture higher-level relationships learned by the Teacher, predicting correlations or dependencies between different features or outputs.

Beyond these types, various algorithms have been developed to implement KD.

DeepSeek has not officially acknowledged using KD with models like GPT-4. Instead, they claim to have used it to transfer knowledge from their R1 model to smaller models like LLaMA and Qwen, thereby enhancing their reasoning capabilities. In practice, since GPT-4 is not an open-source model, it's impossible for DeepSeek to directly utilize it for KD. However, if DeepSeek did employ KD to train their model, they might have done so indirectly, potentially through methods like data collection via APIs. OpenAI provides API access to GPT-4. DeepSeek could have used this API to input large amounts of text into GPT-4 and collected its output to train their own model. Security researchers at Microsoft have reported detecting data exfiltration from an OpenAI developer account linked to DeepSeek.

If DeepSeek did indeed leverage KD to effectively emulate a large LLM, it forces us to consider how the operations of large LLM companies might change in the following aspects:

• Stricter API Access Restrictions: Given the allegations that DeepSeek collected GPT-4 output data via its API, large LLM companies might tighten API access restrictions.
• Output Data Watermarking: Large LLM companies could implement watermarking techniques on the output data provided through their APIs. This would allow them to trace the origin of the output data and prevent unauthorized use.
• Differentiated API Services: To make KD more difficult, LLM companies could segment their API services. For instance, they might offer specialized APIs tailored to specific tasks.
• Development of Anti-KD Techniques: Research could be directed towards adding noise to training data, or other techniques that hinder distillation.
• Strengthened Legal Action: There were reports that OpenAI might take legal action against DeepSeek, indicating a potential escalation in legal responses to perceived KD-based model replication.

This situation highlights the evolving landscape of AI development and the challenges of protecting intellectual property in the age of readily available powerful language models. The interplay between open-source development and proprietary models will likely continue to be a dynamic and contentious area.