{"data":{"capabilities":{"chat":true,"reasoning":true,"tools":true,"vision":false},"description":"Inkling-Small-NVFP4 on two RTX PRO 6000 Blackwell Max-Q (sm_120) via SGLang TP-2, ctx 131,072. Upstream dspark image + ONE patched kernel file (two one-line fixes: grouped-GEMM num_stages 4->3 for sm_120's 101,376 B shared memory; silu_and_mul rerouted from Helion, which has no sm_120 config, to the in-file Triton kernel — verified bit-exact). MoE runner marlin, attention triton, KV fp8_e5m2. Stock vLLM v0.27.1 has no sm_120 attention path for this model and asserts at startup; with a locally ported attention kernel it serves but computes garbage (cutlass NVFP4 MoE wrong at 256 experts/top-6 on sm_120; marlin-in-vLLM did not rescue that stack), so SGLang is the coherent path. Temporary bridge: retire when upstream SGLang ships sm_120 kernels for this model.","engine":{"graph_mode":"full decode capture bs<=4; piecewise CUDA graph disabled","name":"sglang","version":"lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7 + 1 patched file (SGLang commit b7252cc6b0c78b25ecea7ee5efa91a6ae37d0f19); built image local/sglang-inkling:sm120"},"facts":{},"hardware_count":2,"hardware_id":"rtx-pro-6000-blackwell-96gb","id":"inkling-small-nvfp4-rtxpro6000-sglang-tp2","launch":{"accelerator_backend":"nvidia","arguments":["python3","-m","sglang.launch_server","--model-path","thinkingmachines/Inkling-Small-NVFP4","--served-model-name","inkling-small","--tp-size=2","--mem-fraction-static=0.965","--context-length=131072","--max-running-requests=4","--chunked-prefill-size=8192","--attention-backend=triton","--moe-runner-backend=marlin","--kv-cache-dtype=fp8_e5m2","--reasoning-parser=inkling","--tool-call-parser=inkling","--disable-piecewise-cuda-graph","--cuda-graph-max-bs=4","--disable-custom-all-reduce","--trust-remote-code","--host","0.0.0.0","--port","8010"],"container":{"captured_at":"2026-08-27T06:04:13.773Z","compose_file":null,"digest":null,"image":"local/sglang-inkling:sm120 (build from source repo inkling-small-nvfp4/Dockerfile; base lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7)","reason":"image-tag-without-content-digest","runtime":"docker","source":[{"captured_at":"2026-08-27T06:04:13.773Z","kind":"recipe-launch","url":"https://github.com/ppickle1989/sm120-sglang-recipes"}],"state":"mutable"},"container_port":8010,"environment":{"CUDA_DEVICE_ORDER":"PCI_BUS_ID","CUDA_VISIBLE_DEVICES":"0,1","HF_HOME":"/data/huggingface","HF_HUB_OFFLINE":"1","NCCL_IB_DISABLE":"1","NCCL_P2P_DISABLE":"1","NCCL_P2P_LEVEL":"SYS","NCCL_PROTO":"LL,LL128,Simple","NVIDIA_VISIBLE_DEVICES":"${GPUS}","PYTORCH_CUDA_ALLOC_CONF":"expandable_segments:True","SGLANG_INKLING_KEEP_PACKED_TOPK":"1","TORCH_CUDA_ARCH_LIST":"12.0a"},"host_port":8010,"image":"local/sglang-inkling:sm120 (build from source repo inkling-small-nvfp4/Dockerfile; base lmsysorg/sglang:dev-cu13-inkling-dspark@sha256:fbea1a4e25b26660dbc2384a27ead8817e9b7670f257b5c3143e0450d14524d7)","ipc":"host","kind":"docker","launch_command":"inkling-small-nvfp4/run.sh (docker run, foreground)","mounts":[{"read_only":true,"source":"${HF_HOME}","target":"/data/huggingface"},{"read_only":false,"source":"inkling-sgl-cache (named volume)","target":"/root/.cache"}],"shm_size":"32g","source":{"commit":"769f4704362af24e355f4117b60dd6f544ea0580","paths":["inkling-small-nvfp4/Dockerfile","inkling-small-nvfp4/patches/inkling_moe.py","inkling-small-nvfp4/patches/inkling_moe.diff","inkling-small-nvfp4/PROVENANCE.md","inkling-small-nvfp4/run.sh","manifest.json","benchmarks/inkling-small-nvfp4-tp2-tps-20260826.json"],"repository":"https://github.com/ppickle1989/sm120-sglang-recipes"}},"metadata":{"acceptance":{"completion":true,"tools":true,"vision":false},"context_note":"a request that produced a 73,077-token completion (finish=stop) was served at the configured 131,072 context; requests exceeding it return HTTP 400","hardware_variant":"RTX PRO 6000 Blackwell Max-Q Workstation Edition (300 W)","interconnect":"PCIe 5.0, x8 + x4 as reported by nvidia-smi under load (AM5 consumer lane split); host-mediated multi-GPU, no NVLink; NCCL_P2P_DISABLE=1 required on this board","traces":[{"date":"2026-08-25T08:17-05:00","detail":"Stock vLLM v0.27.1 asserts at startup (no sm_120 paged-KV attention for this model). With a locally ported sm_120 attention kernel it loads and serves; positive control FAILS: token salad in both reasoning and content at temp 1.0 / effort none. Marlin MoE swap inside vLLM (three variants) still garbage. Engine fault, not hardware/quant.","event":"vllm_first_light_garbage"},{"date":"2026-08-25","detail":"SGLang TP-2 NCCL init hung 26 min with both GPUs at 100% util (~1 GB) on this AM5 board; NCCL_P2P_DISABLE=1 -> init in 0.89 s.","event":"nccl_init_hang"},{"date":"2026-08-25","detail":"'Required 110592 > 101376' from the small-M grouped-GEMM decode config; num_stages 4->3 (73,728 B) in inkling_moe.py.","event":"fix1_grouped_gemm_smem"},{"date":"2026-08-25","detail":"silu_and_mul_interleaved_sm_120.json missing and cannot be autogenerated (int32/int64 catch-22); interleaved branch rerouted to the in-file pure-Triton silu_and_mul_triton. Later tensor-level diff vs eager reference: bit-exact.","event":"fix2_helion_silu"},{"date":"2026-08-25","detail":"Health 200 in 215 s on first start (110 s on later starts with the kernel cache volume). Positive control PASS: factual Q&A, 17x23=391, tool call parses (inkling parser), thinking effort low/high/off all coherent; ~92-117 tok/s effective.","event":"startup_success"},{"date":"2026-08-25","detail":"21-prompt parity check against the vendor API: 21/21 locally = 21/21 on the vendor API for the same model (0 drop).","event":"canary_parity"},{"date":"2026-08-25T13:35-05:00","detail":"Bind-mount replaced by a built image: Dockerfile asserts the in-image original sha256 (f23a3775...) and the patched sha256 (c101a7e8...) at build time.","event":"managed_image_built"},{"date":"2026-08-25","detail":"ctx 32768 -> 65536 -> 131072; KV pool unchanged at 474,388 fp8_e5m2 tokens (fits 3 concurrent at 128k). bf16 KV pool would be 237,194 tokens; a 24/25 vs 25/25 retrieval probe showed no meaningful fp8-KV cost.","event":"context_raise"},{"date":"2026-08-26T02:14-05:00","detail":"Idle-server tok/s bench: ~115 tok/s effective on every prompt class, decode 114.6-117.9, TTFT 0.06 s; 4 concurrent 306.6 tok/s aggregate (~85/stream, TTFT 0.32 s). Later sustained load (9.6 h contiguous): peaks 88 C / 277-283 W per GPU, no throttling.","event":"benchmark"}]},"model_instance_id":"thinkingmachines-inkling-small-nvfp4--nvfp4","provenance":{"captured_at":"2026-08-27T06:04:13.773Z","sources":[{"captured_at":"2026-08-27T06:04:13.773Z","kind":"normalized-recipe","url":"https://github.com/ppickle1989/sm120-sglang-recipes"}]},"recipe_source":"ppickle1989","schema_version":"local-ai-registry/v1","serving":{"api":"openai/v1","kv_cache_dtype":"fp8_e5m2","kv_cache_tokens":474388,"max_concurrency":4,"max_context_tokens":131072,"tensor_parallel":2},"speed_sweep_ids":["inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep"],"status":"candidate","huggingface":{"link_type":"repository","provenance":{"captured_at":"2026-08-27T06:04:13.773Z","sources":[{"captured_at":"2026-08-27T06:04:13.773Z","kind":"huggingface-api","url":"https://huggingface.co/api/models/thinkingmachines/Inkling-Small-NVFP4"}]},"reason":"hf-api-confirmed-public","repository":"thinkingmachines/Inkling-Small-NVFP4","status":"known","url":"https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4"},"registry":{"launchable":false,"speed_evidence":{"available":true,"count":1,"speed_sweep_ids":["inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep"],"detail_urls":["/api/v1/speed-sweep/inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep"]},"runtime":"docker"},"relationships":{"hardware":{"api":"/api/v1/hardware/rtx-pro-6000-blackwell-96gb","href":"/hardware/rtx-pro-6000-blackwell-96gb","id":"rtx-pro-6000-blackwell-96gb","name":"RTX PRO 6000 Blackwell"},"model":{"api":"/api/v1/models/inkling-small","href":"/models/inkling-small","id":"inkling-small","name":"Inkling-Small"},"model_instance":{"api":"/api/v1/model-instances/thinkingmachines-inkling-small-nvfp4--nvfp4","href":"/model-instances/thinkingmachines-inkling-small-nvfp4--nvfp4","id":"thinkingmachines-inkling-small-nvfp4--nvfp4","name":"thinkingmachines/Inkling-Small-NVFP4"},"speed_sweep":[{"api":"/api/v1/speed-sweep/inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep","href":"/speed-sweep/inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep","id":"inkling-small-nvfp4-rtxpro6000-sglang-tp2-sweep"}]}},"meta":{"source":"registry"}}