Recipe
glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4
glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4Validated GLM-5.3 EXL3/TR3 3.0 bpw recipe on four RTX PRO 6000 Blackwell 96 GB GPUs with TP4/DCP4, MTP3, quantized KV cache, full and piecewise CUDA graphs, an 800k context envelope, and uncapped C1-C3 matched-window measurements.
Record
- Status
- candidate
- Source
- 0xsero
- Engine
- vllm
- Engine version
- 0.26.1rc0+infernal.invocation.cu133.r17.vllmc53cc73.b12xc0a44a1
- Graph
- full-and-piecewise
- Accelerators
- 4
- Tensor parallel
- 4
- Context tokens
- 800,000
- Max concurrency
- 3
- KV cache tokens
- 862,975
- chat
- yes
- reasoning
- yes
- tools
- yes
- vision
- no
Hugging Face model card
Identity
https://huggingface.co/davidsyoung/GLM-5.3-EXL3-TR3-3.0bpw- Repository
- davidsyoung/GLM-5.3-EXL3-TR3-3.0bpw
- Status
- known
- Link type
- Exact Hub repository
Public Hugging Face repository confirmed by the Hub API.
Candidate: useful compatibility or speed evidence. The registry does not offer Run until promotion requirements are met.
Docker configuration
vllm
Container · candidate · docker
Candidate evidence — not a Run contract
- Image
voipmonitor/vllm:infernal-invocation-vllmc53cc73-b12xc0a44a1-fi1ac6942-cu133-torch213-20260817-r17@sha256:c5e96c5bcc5a073f7ce6b56173d88538de3a416900cff97c88b4bf7967fe1dc0- Digest
sha256:c5e96c5bcc5a073f7ce6b56173d88538de3a416900cff97c88b4bf7967fe1dc0- Compose file
docs/notes/glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-compose.yml- Port
- 8000
Observed source tokens
Mechanical split of the source command. Unverified against the engine CLI. The registry does not offer Run for this recipe.
--default-chat-template-kwargs{"reasoning_effort":"none"}--attention-config{"indexer_kv_dtype":"fp8"}--kv-cache-memory-bytes8589934592
| Flag | Value |
|---|---|
--default-chat-template-kwargs | {"reasoning_effort":"none"} |
--attention-config | {"indexer_kv_dtype":"fp8"} |
--kv-cache-memory-bytes | 8589934592 |
Environment
| Variable | Value |
|---|---|
ASYNC_SCHEDULING | 0 |
B12X_PCIE_DMA | 0 |
CUDA_DEVICE_MAX_CONNECTIONS | 1 |
DCP | 4 |
DCP_A2A_LARGE_BACKEND | ag_rs |
DCP_BACKEND | a2a |
DCP_CKV_GATHER_MAX_TOKENS | 262144 |
DCP_CKV_PREFETCH_DEPTH | 0 |
DCP_CKV_PREFETCH_WORKSPACE_MIB | 0 |
F8_DMA | 0 |
GPU_MEMORY_UTILIZATION | 0.98 |
GRAPH | 12 |
INSTANTTENSOR_BACKEND | BUFFERED |
INSTANTTENSOR_COPY | 0 |
KV_CACHE_DTYPE | nvfp4_ds_mla |
KV_FP8_ROPE | 0 |
LMCACHE_MODE | off |
LOAD_FORMAT | instanttensor |
MAX_BATCHED_TOKENS | 3072 |
MAX_MODEL_LEN | 800000 |
MAX_NUM_SEQS | 3 |
MODEL | /model |
MODEL_FAMILY | glm52-exl3 |
MOE_BACKEND | b12x |
MOE_MODE | a16 |
MTP | 3 |
MTP_DRAFT_SAMPLE_METHOD | greedy |
MTP_MOE_BACKEND | triton |
PCIE_CALIBRATION | off |
QUANTIZATION | exl3 |
SERVED_MODEL_NAME | glm-5.3 |
TP | 4 |
VLLM_B12X_GLM_NOPE_NVFP4 | 1 |
VLLM_ENGINE_READY_TIMEOUT_S | 3600 |
VLLM_EXL3_PREFILL_CAPACITY | 2048 |
VLLM_EXL3_PREFILL_CHUNK | 128 |
VLLM_EXL3_PREFILL_TRELLIS | 1 |
VLLM_EXL3_TRELLIS_BLOCK_M | 8 |
VLLM_EXL3_TRELLIS_MAX_M | 16 |
VLLM_EXL3_TRELLIS_MIN_M | 1 |
VLLM_NVFP4_MLA_DYNAMIC_SCALE | 0 |
Mounts
| Source | Target |
|---|---|
${MODEL_ROOT}/GLM-5.3-EXL3-TR3-3.0bpw | /model (read-only) |
${MODEL_ROOT}/GLM-5.3-EXL3-TR3-3.0bpw | /models (read-only) |
Measured speed
| Concurrency | Context | Prefill | Decode | TTFT ms | Status | Sweep |
|---|---|---|---|---|---|---|
| 1 | 78 | 1,752.2 | 61.2 | 325.5 | validated | glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4-sweep |
| 2 | 78 | 1,768.1 | 42.9 | 503.1 | validated | glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4-sweep |
| 3 | 78 | 1,855.5 | 38.5 | 675 | validated | glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4-sweep |
Remaining fields
Identity, launch, related records, and measured speed are shown above. This is the rest of the normalized record.
- hardware count
- 4
- hardware id
- rtx-pro-6000-blackwell-96gb
- id
- glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4
- model instance id
- davidsyoung-glm-5-3-exl3-tr3-3-0bpw--3-0-bpw
- recipe source
- 0xsero
- schema version
- local-ai-registry/v1
- speed sweep ids
- glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-96gb-vllm-tp4-sweep
- status
- candidate
capabilities
- chat
- Yes
- reasoning
- Yes
- tools
- Yes
- vision
- No
engine
- graph mode
- full-and-piecewise
- name
- vllm
- version
- 0.26.1rc0+infernal.invocation.cu133.r17.vllmc53cc73.b12xc0a44a1
serving
- data context parallel
- 4
- explicit kv cache bytes per rank
- 8,589,934,592
- kv cache tokens
- 862,975
- max concurrency
- 3
- max context tokens
- 800,000
- max num batched tokens
- 3,072
- max num seqs
- 3
- tensor parallel
- 4
Provenance & metadata (3)
facts
- capabilities.chat · provenance · captured at
- 2026-09-01T04:00:19Z
capabilities.chat · reason natural-stop-chat-completion-passed
capabilities.chat · state known
- capabilities.reasoning · provenance · captured at
- 2026-09-01T04:00:19Z
capabilities.reasoning · reason official-reasoning-effort-support-and-live-low-effort-natural-stop
capabilities.reasoning · state known
- capabilities.tools · provenance · captured at
- 2026-09-01T04:00:19Z
capabilities.tools · reason required-function-call-produced-correct-tool-name-and-arguments
capabilities.tools · state known
- capabilities.vision · provenance · captured at
- 2026-09-01T04:00:19Z
capabilities.vision · reason language-model-only-glm-moe-dsa-architecture
capabilities.vision · state known
- engine.graph mode · provenance · captured at
- 2026-09-01T04:00:19Z
engine.graph mode · reason runtime-captured-full-and-piecewise-cuda-graphs
engine.graph mode · state known
- engine.version · provenance · captured at
- 2026-09-01T04:00:19Z
engine.version · reason live-vllm-version-endpoint
engine.version · state known
- serving.kv cache tokens · provenance · captured at
- 2026-09-01T04:00:19Z
serving.kv cache tokens · reason live-runtime-kv-pool-capacity
serving.kv cache tokens · state known
- serving.max concurrency · provenance · captured at
- 2026-09-01T04:00:19Z
serving.max concurrency · reason uncapped-c3-sustained-decode-completed-with-effective-concurrency-3-and-no-queue
serving.max concurrency · state known
- serving.max context tokens · provenance · captured at
- 2026-09-01T04:00:19Z
serving.max context tokens · reason live-served-model-context-envelope
serving.max context tokens · state known
metadata
- acceptance evidence
- docs/notes/glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-20260901.json
- quality sidecars · arithmetic
- Yes
- quality sidecars · reasoning request
- Yes
- quality sidecars · structured json
- Yes
- quality sidecars · tool call
- Yes
- runtime identity · architecture
- GlmMoeDsaForCausalLM
- runtime identity · cuda graphs
- full-and-piecewise
- runtime identity · indexer kv dtype
- fp8
- runtime identity · kv cache dtype
- nvfp4_ds_mla
- runtime identity · load format
- instanttensor
- runtime identity · parallelism
- tp4-dcp4
- runtime identity · quantization
- exl3
- runtime identity · speculation
- mtp3
- scheduler · kv cache memory bytes per rank
- 8,589,934,592
- scheduler · max num batched tokens
- 3,072
- scheduler · max num seqs
- 3
provenance
- captured at
- 2026-09-01T04:00:19Z
sources
| captured at | kind | url |
|---|---|---|
| 2026-09-01T04:00:19Z | validated-runtime-recipe | github.com/0xSero/local-ai-registry/blob/main/docs/notes/glm-5-3-exl3-tr3-3-0bpw-rtx-pro-6000-blackwell-20260901.json ↗ |