Dots.LLM1: 142B MoE Model Trained on 11.2T Real-World Tokens

Developed by the rednote-hilab team, Dots.LLM1 is a large-scale sparse model featuring 142 billion total parameters. By activating only 14 billion parameters during inference, the model minimizes operational costs without compromising output quality. The model weights are currently available on Hugging Face.

Architecture and Training

Model Type: Dots.LLM1 utilizes a Mixture of Experts (MoE) architecture. Of its 142B total parameters, only 14B are activated per token. The model was trained on 11.2 trillion tokens of curated, real-world text, notably excluding synthetic data during the pretraining phase.

Training Phases: The development process involved comprehensive pretraining followed by supervised fine-tuning (SFT).

Network Details:

• QK-Norm: Applied within the attention layers to stabilize training.
• Fine-grained MoE Routing: Out of a pool of 128 experts, the model selects the top 6 experts per token, supplemented by 2 shared experts to maintain foundational knowledge.

Core Specifications:

• Layers & Heads: 62 layers and 32 attention heads.
• Bilingual Capabilities: Full support for Chinese and English with a 32,768-token context window.
• Licensing: Released under the MIT license.

Three-Tier Data Framework: The team implemented a specialized pipeline to generate diverse, high-quality data at scale. This framework ensures that pretraining relies entirely on authentic human-generated content rather than synthetic shortcuts.

Performance and Efficiency: By activating only 14 billion parameters, the model significantly reduces computational overhead. Dots.LLM1 delivers performance comparable to Qwen2.5-72B while requiring substantially less power and hardware resources.

Infrastructure Optimization: The project employs several techniques to maximize hardware utilization, including overlapping communication and computation during the MoE all-to-all step. It also utilizes an interleaved 1F1B pipeline schedule and optimized grouped GEMM kernels.

Open Research: To support the research community, intermediate checkpoints from every trillion tokens of training have been made public, providing a transparent view of the model's learning trajectory.

Model Downloads

Docker Deployment (Recommended)

To deploy the model, pull the official image from Docker Hub and initialize the server:

docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    rednotehilab/dots1:vllm-openai-v0.9.0.1 \
    --model rednote-hilab/dots.llm1.inst \
    --tensor-parallel-size 8 \
    --trust-remote-code \
    --served-model-name dots1

Verify the deployment with a test request:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "dots1",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ],
        "max_tokens": 32,
        "temperature": 0
    }'

Hugging Face Inference Examples

Text Generation

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "rednote-hilab/dots.llm1.base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)

text = "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

Chat Completion

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "rednote-hilab/dots.llm1.inst"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)

messages = [{"role": "user", "content": "Write a piece of quicksort code in C++"}]
input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(input_tensor.to(model.device), max_new_tokens=200)
result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
print(result)

Other Inference Tools

vLLM: Deploy an OpenAI-compatible API endpoint.

vllm serve rednote-hilab/dots.llm1.inst --port 8000 --tensor-parallel-size 8

SGLang: A lightweight serving option that can be launched with a single command.

python -m sglang.launch_server --model-path rednote-hilab/dots.llm1.inst --tp 8 --host 0.0.0.0 --port 8000

Backed by 11.2 trillion tokens of authentic data, Dots.LLM1 competes directly with top-tier models like Qwen2.5-72B. It offers a performance-to-cost ratio that makes it a standout choice within the open-source ecosystem.