Local AI Registry

Recipe

deepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2

deepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2
JSON APIFind compatible recipes

DeepSeek-V4-Flash-0731 (vendor fp8 dense + fp4 experts, served stock) on two RTX PRO 6000 Blackwell Max-Q (sm_120) via the upstream SGLang dspark image, TP-2, ctx 131,072, thinking mode. No patches. Weights 74.9 GB per rank; MLA KV pool 1,722,368 tokens. The one trap: SGLang's DeepSeek-V4 template defaults to CHAT mode (zero reasoning tokens) — send chat_template_kwargs {"thinking": true} per request to match the vendor API's posture; enable_thinking is ignored and reasoning_effort only adds a prompt prefix.

Record

Status
candidate
Source
ppickle1989
Engine
sglang
Engine version
lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7 (stock; SGLang commit b7252cc6b0c78b25ecea7ee5efa91a6ae37d0f19); no patches
Graph
full decode capture bs<=4; piecewise CUDA graph disabled
Accelerators
2
Tensor parallel
2
Context tokens
131,072
Max concurrency
4
KV cache tokens
1,722,368
chat
yes
reasoning
yes
tools
yes
vision
no

Hugging Face model card

Identity

https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
Repository
deepseek-ai/DeepSeek-V4-Flash-0731
Status
known
Link type
Exact Hub repository

Public Hugging Face repository confirmed by the Hub API.

Candidate: useful compatibility or speed evidence. The registry does not offer Run until promotion requirements are met.

Docker configuration

sglang

Container · candidate · docker

Candidate evidence — not a Run contract

Image
lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7
Digest
sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7
Port
8011

Observed source tokens

Mechanical split of the source command. Unverified against the engine CLI. The registry does not offer Run for this recipe.

  1. python3
  2. -m
  3. sglang.launch_server
  4. --model-path
  5. deepseek-ai/DeepSeek-V4-Flash-0731
  6. --served-model-name
  7. deepseek-v4-flash
  8. --tp-size=2
  9. --mem-fraction-static=0.93
  10. --context-length=131072
  11. --max-running-requests=4
  12. --chunked-prefill-size=8192
  13. --attention-backend=triton
  14. --reasoning-parser=deepseek-v4
  15. --tool-call-parser=deepseekv4
  16. --disable-piecewise-cuda-graph
  17. --cuda-graph-max-bs=4
  18. --disable-custom-all-reduce
  19. --trust-remote-code
  20. --host
  21. 0.0.0.0
  22. --port
  23. 8011
FlagValue
-msglang.launch_server
--model-pathdeepseek-ai/DeepSeek-V4-Flash-0731
--served-model-namedeepseek-v4-flash
--tp-size2
--mem-fraction-static0.93
--context-length131072
--max-running-requests4
--chunked-prefill-size8192
--attention-backendtriton
--reasoning-parserdeepseek-v4
--tool-call-parserdeepseekv4
--cuda-graph-max-bs4
--host0.0.0.0
--port8011

Environment

VariableValue
CUDA_DEVICE_ORDERPCI_BUS_ID
CUDA_VISIBLE_DEVICES0,1
HF_HOME/data/huggingface
HF_HUB_OFFLINE1
NCCL_IB_DISABLE1
NCCL_P2P_DISABLE1
NCCL_P2P_LEVELSYS
NCCL_PROTOLL,LL128,Simple
NVIDIA_VISIBLE_DEVICES${GPUS}
PYTORCH_CUDA_ALLOC_CONFexpandable_segments:True
TORCH_CUDA_ARCH_LIST12.0a

Mounts

SourceTarget
${HF_HOME}/data/huggingface (read-only)
dsv4-sgl-cache (named volume)/root/.cache

Measured speed

ConcurrencyContextPrefillDecodeTTFT msStatusSweep
130—57.980measureddeepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2-sweep
140—57.884measureddeepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2-sweep
140—57.980measureddeepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2-sweep
150—57.880measureddeepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2-sweep
440—47.2364measureddeepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2-sweep

Remaining fields

Identity, launch, related records, and measured speed are shown above. This is the rest of the normalized record.

hardware count
2
hardware id
rtx-pro-6000-blackwell-96gb
id
deepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2
model instance id
deepseek-ai-deepseek-v4-flash-0731--fp8
recipe source
ppickle1989
schema version
local-ai-registry/v1
speed sweep ids
deepseek-v4-flash-0731-fp8-fp4-rtxpro6000-sglang-tp2-sweep
status
candidate

capabilities

chat
Yes
reasoning
Yes
tools
Yes
vision
No

engine

graph mode
full decode capture bs<=4; piecewise CUDA graph disabled
name
sglang
version
lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7 (stock; SGLang commit b7252cc6b0c78b25ecea7ee5efa91a6ae37d0f19); no patches

serving

api
openai/v1
kv cache dtype
fp8_e4m3 (auto)
kv cache tokens
1,722,368
max concurrency
4
max context tokens
131,072
tensor parallel
2
Provenance & metadata (3)

facts

metadata

acceptance · completion
Yes
acceptance · tools
Yes
acceptance · vision
No
context note
a request that produced a 128,000-token completion (finish=length) was served at the configured 131,072 context, i.e. the full window is usable; requests exceeding it return HTTP 400
hardware variant
RTX PRO 6000 Blackwell Max-Q Workstation Edition (300 W)
interconnect
PCIe 5.0, x8 + x4 as reported by nvidia-smi under load (AM5 consumer lane split); host-mediated multi-GPU, no NVLink; NCCL_P2P_DISABLE=1 required on this board

traces

datedetailevent
2026-08-25Upstream dspark image serves the model with no patches: weights load 46 s, 74.92 GB/rank; health in ~2 min when kernels are cached, up to ~10 min on a cold first start.startup_success_stock
2026-08-25First probe launched without --reasoning-parser/--tool-call-parser: tool calls could not parse. Relaunched with --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 (names from the in-image ReasoningParser.DetectorMap / ToolCallParserEnum); positive control then PASS incl. parsed tool call.gotcha_parser_flags
2026-08-25SGLang dsv4 defaults to chat mode: 0 reasoning tokens unless chat_template_kwargs {"thinking": true} is sent per request (enable_thinking is ignored; reasoning_effort only adds a prompt prefix). Vendor API serves in thinking mode, so parity needs the kwarg.gotcha_chat_mode_default
2026-08-2521-prompt parity check against the vendor API with thinking:true: 21/21 locally = 21/21 on the vendor API.canary_parity
2026-08-26T12:54-05:00ctx 65536 -> 131072 (mem-fraction-static 0.93): max_total_num_tokens=1,722,368, available_gpu_mem 6.0 GB after pools. Long-generation probes (primes to 4000, 600-digit binary, 40x40 table, 800 squares at a 32k budget) all 0 errors; a 12k budget was exhausted inside reasoning on the enumeration task (budget, not drift).context_raise
2026-08-26T12:56-05:00Idle-server tok/s bench (thinking:true): 56-58 tok/s effective on every prompt class, decode 57.8, TTFT 0.08 s; 4 concurrent 155.2 tok/s aggregate (46.5-48.2/stream, TTFT 0.35-0.42 s). Flat across output length.benchmark
2026-08-26Sustained load window 2026-08-26 (17 h, GPUs >= 50 % busy for 16.7 h, longest contiguous span 9.6 h): GPU peaks 88 C / 277-283 W (300 W limit), CPU package 85 C, VRM 57 C, NVMe 59 C; no throttling.thermal

provenance

captured at
2026-08-27T06:04:13.773Z

sources

captured atkindurl
2026-08-27T06:04:13.773Znormalized-recipegithub.com/ppickle1989/sm120-sglang-recipes ↗