Kimi K3 Hub

How to Deploy Kimi K3 Locally: vLLM, SGLang, and hardware reality

Jul 28, 2026

Before you deploy Kimi K3 locally, define what "local" means. For Kimi K3, local deployment means self-hosting on a Linux multi-GPU server or private GPU cluster. It does not mean running the full model comfortably on a laptop or a single consumer GPU.

If you only want to use the model through an API, start with the Kimi K3 guide hub and the API setup guides on this site. This article is for teams that need private inference, internal networking, benchmark control, or a custom serving stack.

Decide whether local deployment is the right path

Moonshot's public materials describe Kimi K3 as a 2.8T-parameter MoE model with 104B activated parameters, a 1,048,576-token context window, MXFP4 weights, and MXFP8 activations. That is the first fact to internalize: this is not a small open-weight model you casually run after a download.

Ask three questions before you provision machines:

  1. Do you have a multi-GPU server with enough memory, interconnect, host RAM, and storage?
  2. Do you truly need local weights instead of the official Kimi API?
  3. Can your team maintain inference images, engine versions, monitoring, and upgrades?

If the answers are uncertain, validate the model through the official API first. Move to self-hosting only when privacy, latency, cost modeling, internal access, or inference customization makes it necessary.

The official Kimi K3 README currently points to vLLM, SGLang, and TokenSpeed for inference. Choose the engine by workflow rather than taste:

  • vLLM: best first stop for OpenAI-compatible serving, batching, prefix caching, and production-oriented LLM infrastructure.
  • SGLang: useful when agent serving, tool calls, structured output, or fine-grained scheduling matter.
  • TokenSpeed: a better fit for teams already exploring high-performance serving and cluster-level tuning around TokenSpeed.

As of July 28, 2026, the safest path is to start from the official recipe for the engine you choose. Use the current image, flags, and known limitations from that recipe instead of copying an old command from memory.

Prepare the server

A realistic Kimi K3 local deployment needs these pieces:

  • A Linux GPU server or containerized GPU cluster.
  • NVIDIA or AMD GPU drivers that match the serving image.
  • Enough GPU memory, host memory, local SSD, or fast shared storage.
  • Access to the Hugging Face model repository or another official weight source.
  • Docker, NVIDIA Container Toolkit, or an equivalent container runtime.
  • An internal service port such as 8000 or 30000.

Start by preparing environment variables:

export HF_TOKEN="YOUR_HUGGING_FACE_TOKEN"
export MODEL_ID="moonshotai/Kimi-K3"
mkdir -p ~/.cache/huggingface

The weights, download time, and validation time are all large. In production, keep the model cache on persistent storage and avoid multiple containers redownloading the same weights.

Path 1: deploy Kimi K3 with vLLM

vLLM is the most common first evaluation path because it can expose an OpenAI-compatible /v1/chat/completions endpoint. That makes it easier to connect existing apps, gateways, and benchmark scripts.

Start from the official vLLM Kimi K3 recipe and use the image and flags it specifies. The launch shape looks like this:

docker run --gpus all \
  --ipc=host \
  --shm-size=32g \
  -p 8000:8000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  <VLLM_KIMI_K3_IMAGE_FROM_OFFICIAL_RECIPE> \
  vllm serve "$MODEL_ID" \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --max-model-len 1048576

Do not casually replace <VLLM_KIMI_K3_IMAGE_FROM_OFFICIAL_RECIPE> with an older generic vLLM image. Kimi K3 serving depends on fast-moving support for quantization, attention kernels, parallelism, and parsing behavior.

For a small smoke test, lower --max-model-len to 32768 or 65536 first. Confirm startup, response quality, token accounting, and logs before attempting full 1M-token serving.

Path 2: deploy Kimi K3 with SGLang

SGLang is another engine named by the official Kimi K3 README. It is worth evaluating when your serving layer needs richer agent flows, multi-turn control, structured output, tool handling, or scheduling behavior.

A minimal service shape looks like this:

docker run --gpus all \
  --ipc=host \
  --shm-size=32g \
  -p 30000:30000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path "$MODEL_ID" \
    --host 0.0.0.0 \
    --port 30000

For production, this minimal command is not enough. Use the official SGLang cookbook to set tensor parallelism, expert parallelism, data parallelism, memory fraction, max running requests, and parser settings for your hardware.

Path 3: evaluate TokenSpeed

TokenSpeed is also listed by the official Kimi K3 README as a supported inference option. It is more relevant for teams already working on high-performance serving experiments or specialized cluster tuning.

If you choose TokenSpeed, start from the official TokenSpeed recipes. Do not copy vLLM or SGLang flags into TokenSpeed by analogy. Parallelism, caching, scheduling, and model loading are engine-specific.

Test the OpenAI-compatible endpoint

After the server starts, test it before connecting real traffic:

curl http://localhost:8000/v1/chat/completions \
  -H "content-type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K3",
    "messages": [
      {
        "role": "user",
        "content": "List three things teams often miss when they deploy Kimi K3 locally."
      }
    ],
    "max_tokens": 512,
    "reasoning_effort": "low"
  }'

If you are using SGLang on port 30000, change the URL accordingly.

The official Kimi K3 README notes that the model returns reasoning_content and that multi-turn conversations or tool calls should pass the full assistant message back into messages, including reasoning_content and tool_calls. Keeping only content can break later turns.

Production checklist

Before routing users to the local service, verify these items:

  • Model weights and image versions are pinned and reproducible.
  • GPU memory, KV cache, queue length, throughput, and errors are monitored.
  • The API layer has rate limits, timeouts, retry policy, and request-size limits.
  • Logs capture request IDs, input tokens, output tokens, latency, and error type.
  • Long-context requests have explicit quotas instead of defaulting everyone to 1M tokens.
  • Multi-turn conversations preserve the complete assistant message, including reasoning_content.
  • OpenAI-compatible clients use the correct base URL and model name.

Local deployment gives you control. It also makes you responsible for capacity planning, failure recovery, security, and upgrades. Do not stop at "it returned text once"; test concurrency, long context, failure handling, and rollback.

Common questions

Can I deploy Kimi K3 locally on a normal desktop?

No, not in a practical way. Kimi K3 is sized for multi-GPU servers or clusters. A normal desktop is better suited to calling the official API.

Can I test it on one GPU?

Usually not meaningfully. MXFP4 weights help, but the model scale, KV cache, and serving overhead still exceed a single consumer GPU setup.

Should I choose vLLM or SGLang?

Start with vLLM if you already have OpenAI-compatible gateways, evaluation scripts, and serving workflows. Evaluate SGLang when agent serving or SGLang-specific control matters.

Why does this guide not pin a fixed image name?

Because Kimi K3 serving support is moving quickly. Directing readers to the current official recipe is safer than freezing an image name that can age badly.

References

Kimi K3 Team

How to Deploy Kimi K3 Locally: vLLM, SGLang, and hardware reality | Kimi K3 Blog