DeepSeek LLM Technical Overview

From www.iamdave.ai

DeepSeek AI: The Next Big Breakthrough in AI | DaveAI

DeepSeek is a Chinese AI startup that has made significant strides in the development of large language models (LLMs). The company, founded in May 2023, released its latest model, DeepSeek V3, in December 2024, followed by the reasoning model R1 in January 2025. Here’s a detailed technical overview of the key components and innovations in the DeepSeek LLM series:

DeepSeek-LLM: Laying the Foundation

The initial work, DeepSeek-LLM, focused on understanding the optimal balance between model size and data quality. Key points include:

• Scale Measurement: Instead of measuring scale by the number of parameters, they used non-embedding FLOPs/token to predict computation.
• Training Instability: Addressed through careful tuning of hyperparameters and HPC co-design.
• Data Quality: Higher-quality data can justify a larger model for the same number of tokens.
• Results: DeepSeek-LLM67B outperformed LLaMA-2 70B on math and coding tasks.

DeepSeek-V2: Multi-Head Latent Attention & MoE

DeepSeek-V2 introduced two key innovations to reduce memory and computational overhead:

• Multi-Head Latent Attention (MLA): Compresses Key/Value vectors to reduce memory usage.
• DeepSeekMoE: A sparse Mixture-of-Experts approach that activates a fraction of the feed-forward capacity per token.
• Training & Outcomes: DeepSeek-V2, with ~236B total parameters, was pre-trained on 8.1T tokens. It demonstrated faster and cheaper inference and training while maintaining stability at scale.

DeepSeek-V3: HPC Co-Design

Building on V2, DeepSeek-V3 further extended sparse models to 671B parameters (37B activated) and was trained on 14.8T tokens. Key innovations include:

• Refined MLA: Dynamic low-rank projection, adaptive query compression, improved RoPE handling, joint KV storage, and layer-wise adaptive cache.
• Refined DeepSeekMoE: Auxiliary-loss-free strategy, more activated experts, and higher stability.
• Co-Designed Frameworks: FP8 mixed precision training, DualPipe parallelism, and PTX-level optimizations.
• Training Efficiency: Achieved with 2.788M H800 GPU hours and $5.576M in training costs.

DeepSeek-R1: Reinforcement Learning for Deeper Reasoning

DeepSeek-R1 focuses on enhancing reasoning capabilities through reinforcement learning (RL):

• Emergent Reasoning Behaviors: Self-verification, extended chain-of-thought, exploratory reasoning, and reflection.
• Group Relative Policy Optimization (GRPO): Uses rule-based rewards for accuracy and format.
• Refined Reasoning: Combines small cold-start data, RL, and rejection sampling to improve overall user-friendliness.

Evaluation and Performance

Comprehensive evaluations show that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models in various benchmarks, including:

• Knowledge: Superior performance on MMLU, MMLU-Pro, and GPQA.
• Code, Math, and Reasoning: State-of-the-art performance on math-related and coding benchmarks.

Future Directions

DeepSeek continues to push the boundaries of LLMs by focusing on:

• Innovative Load Balancing: Minimizing performance degradation from load balancing efforts.
• Multi-Token Prediction: Enhancing model performance and inference acceleration.
• FP8 Mixed Precision Training: Reducing GPU memory usage and accelerating training.

For more detailed information, you can refer to the DeepSeek-V3 Technical Report.