Local AI Registry

Recipe

inkling-small-nvfp4-rtxpro6000-sglang-tp2

inkling-small-nvfp4-rtxpro6000-sglang-tp2
JSON APIFind compatible recipes

Inkling-Small-NVFP4 on two RTX PRO 6000 Blackwell Max-Q (sm_120) via SGLang TP-2, ctx 131,072. Upstream dspark image + ONE patched kernel file (two one-line fixes: grouped-GEMM num_stages 4->3 for sm_120's 101,376 B shared memory; silu_and_mul rerouted from Helion, which has no sm_120 config, to the in-file Triton kernel — verified bit-exact). MoE runner marlin, attention triton, KV fp8_e5m2. Stock vLLM v0.27.1 has no sm_120 attention path for this model and asserts at startup; with a locally ported attention kernel it serves but computes garbage (cutlass NVFP4 MoE wrong at 256 experts/top-6 on sm_120; marlin-in-vLLM did not rescue that stack), so SGLang is the coherent path. Temporary bridge: retire when upstream SGLang ships sm_120 kernels for this model.

Record

Status
candidate
Source
ppickle1989
Engine
sglang
Engine version
lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7 + 1 patched file (SGLang commit b7252cc6b0c78b25ecea7ee5efa91a6ae37d0f19); built image local/sglang-inkling:sm120
Graph
full decode capture bs<=4; piecewise CUDA graph disabled
Accelerators
2
Tensor parallel
2
Context tokens
131,072
Max concurrency
4
KV cache tokens
474,388
chat
yes
reasoning
yes
tools
yes
vision
no

Hugging Face model card

Identity

https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4
Repository
thinkingmachines/Inkling-Small-NVFP4
Status
known
Link type
Exact Hub repository

Public Hugging Face repository confirmed by the Hub API.

Candidate: useful compatibility or speed evidence. The registry does not offer Run until promotion requirements are met.

Docker configuration

sglang

Container · candidate · docker

Candidate evidence — not a Run contract

Image
local/sglang-inkling:sm120 (build from source repo inkling-small-nvfp4/Dockerfile; base lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7)
Port
8010

Observed source tokens

Mechanical split of the source command. Unverified against the engine CLI. The registry does not offer Run for this recipe.

  1. python3
  2. -m
  3. sglang.launch_server
  4. --model-path
  5. thinkingmachines/Inkling-Small-NVFP4
  6. --served-model-name
  7. inkling-small
  8. --tp-size=2
  9. --mem-fraction-static=0.965
  10. --context-length=131072
  11. --max-running-requests=4
  12. --chunked-prefill-size=8192
  13. --attention-backend=triton
  14. --moe-runner-backend=marlin
  15. --kv-cache-dtype=fp8_e5m2
  16. --reasoning-parser=inkling
  17. --tool-call-parser=inkling
  18. --disable-piecewise-cuda-graph
  19. --cuda-graph-max-bs=4
  20. --disable-custom-all-reduce
  21. --trust-remote-code
  22. --host
  23. 0.0.0.0
  24. --port
  25. 8010
FlagValue
-msglang.launch_server
--model-paththinkingmachines/Inkling-Small-NVFP4
--served-model-nameinkling-small
--tp-size2
--mem-fraction-static0.965
--context-length131072
--max-running-requests4
--chunked-prefill-size8192
--attention-backendtriton
--moe-runner-backendmarlin
--kv-cache-dtypefp8_e5m2
--reasoning-parserinkling
--tool-call-parserinkling
--cuda-graph-max-bs4
--host0.0.0.0
--port8010

Environment

VariableValue
CUDA_DEVICE_ORDERPCI_BUS_ID
CUDA_VISIBLE_DEVICES0,1
HF_HOME/data/huggingface
HF_HUB_OFFLINE1
NCCL_IB_DISABLE1
NCCL_P2P_DISABLE1
NCCL_P2P_LEVELSYS
NCCL_PROTOLL,LL128,Simple
NVIDIA_VISIBLE_DEVICES${GPUS}
PYTORCH_CUDA_ALLOC_CONFexpandable_segments:True
SGLANG_INKLING_KEEP_PACKED_TOPK1
TORCH_CUDA_ARCH_LIST12.0a

Mounts

SourceTarget
${HF_HOME}/data/huggingface (read-only)
inkling-sgl-cache (named volume)/root/.cache

Measured speed

ConcurrencyContextPrefillDecodeTTFT msStatusSweep
130—117.860measuredinkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep
140—11660measuredinkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep
140—115.663measuredinkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep
150—115.262measuredinkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep
440—84.2320measuredinkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep

Remaining fields

Identity, launch, related records, and measured speed are shown above. This is the rest of the normalized record.

hardware count
2
hardware id
rtx-pro-6000-blackwell-96gb
id
inkling-small-nvfp4-rtxpro6000-sglang-tp2
model instance id
thinkingmachines-inkling-small-nvfp4--nvfp4
recipe source
ppickle1989
schema version
local-ai-registry/v1
speed sweep ids
inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep
status
candidate

capabilities

chat
Yes
reasoning
Yes
tools
Yes
vision
No

engine

graph mode
full decode capture bs<=4; piecewise CUDA graph disabled
name
sglang
version
lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7 + 1 patched file (SGLang commit b7252cc6b0c78b25ecea7ee5efa91a6ae37d0f19); built image local/sglang-inkling:sm120

serving

api
openai/v1
kv cache dtype
fp8_e5m2
kv cache tokens
474,388
max concurrency
4
max context tokens
131,072
tensor parallel
2
Provenance & metadata (3)

facts

metadata

acceptance · completion
Yes
acceptance · tools
Yes
acceptance · vision
No
context note
a request that produced a 73,077-token completion (finish=stop) was served at the configured 131,072 context; requests exceeding it return HTTP 400
hardware variant
RTX PRO 6000 Blackwell Max-Q Workstation Edition (300 W)
interconnect
PCIe 5.0, x8 + x4 as reported by nvidia-smi under load (AM5 consumer lane split); host-mediated multi-GPU, no NVLink; NCCL_P2P_DISABLE=1 required on this board

traces

datedetailevent
2026-08-25T08:17-05:00Stock vLLM v0.27.1 asserts at startup (no sm_120 paged-KV attention for this model). With a locally ported sm_120 attention kernel it loads and serves; positive control FAILS: token salad in both reasoning and content at temp 1.0 / effort none. Marlin MoE swap inside vLLM (three variants) still garbage. Engine fault, not hardware/quant.vllm_first_light_garbage
2026-08-25SGLang TP-2 NCCL init hung 26 min with both GPUs at 100% util (~1 GB) on this AM5 board; NCCL_P2P_DISABLE=1 -> init in 0.89 s.nccl_init_hang
2026-08-25'Required 110592 > 101376' from the small-M grouped-GEMM decode config; num_stages 4->3 (73,728 B) in inkling_moe.py.fix1_grouped_gemm_smem
2026-08-25silu_and_mul_interleaved_sm_120.json missing and cannot be autogenerated (int32/int64 catch-22); interleaved branch rerouted to the in-file pure-Triton silu_and_mul_triton. Later tensor-level diff vs eager reference: bit-exact.fix2_helion_silu
2026-08-25Health 200 in 215 s on first start (110 s on later starts with the kernel cache volume). Positive control PASS: factual Q&A, 17x23=391, tool call parses (inkling parser), thinking effort low/high/off all coherent; ~92-117 tok/s effective.startup_success
2026-08-2521-prompt parity check against the vendor API: 21/21 locally = 21/21 on the vendor API for the same model (0 drop).canary_parity
2026-08-25T13:35-05:00Bind-mount replaced by a built image: Dockerfile asserts the in-image original sha256 (f23a3775...) and the patched sha256 (c101a7e8...) at build time.managed_image_built
2026-08-25ctx 32768 -> 65536 -> 131072; KV pool unchanged at 474,388 fp8_e5m2 tokens (fits 3 concurrent at 128k). bf16 KV pool would be 237,194 tokens; a 24/25 vs 25/25 retrieval probe showed no meaningful fp8-KV cost.context_raise
2026-08-26T02:14-05:00Idle-server tok/s bench: ~115 tok/s effective on every prompt class, decode 114.6-117.9, TTFT 0.06 s; 4 concurrent 306.6 tok/s aggregate (~85/stream, TTFT 0.32 s). Later sustained load (9.6 h contiguous): peaks 88 C / 277-283 W per GPU, no throttling.benchmark

provenance

captured at
2026-08-27T06:04:13.773Z

sources

captured atkindurl
2026-08-27T06:04:13.773Znormalized-recipegithub.com/ppickle1989/sm120-sglang-recipes ↗