Local AI Registry

Recipe

glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2

glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2
JSON APIFind compatible recipes

Publisher-qualified language-only GLM-5.3 Flash TR3 EXL3 4bpw recipe for two RTX PRO 6000 Blackwell GPUs. Uses TP2/EP2/DCP2, B12X sparse MLA, NVFP4 MLA KV, external DFlash2 with seven draft tokens, and full-and-piecewise CUDA graphs. Speed evidence uses the model card's sustained streaming methodology; C2/C4 rows are intentionally omitted because the card marks them capacity-limited rather than throughput claims.

Record

Status
candidate
Source
0xsero
Engine
vllm
Engine version
0.1.dev20111+g7f1e92bec.d20260827
Graph
full-and-piecewise CUDA graphs; DFlash2 full graph capture
Accelerators
2
Tensor parallel
2
Context tokens
98,304
Max concurrency
4
KV cache tokens
151,052
chat
yes
reasoning
yes
tools
yes
vision
no

Hugging Face model card

Identity

https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw
Repository
brandonmusic/GLM-5.3-Flash-tr3-4bpw
Status
known
Link type
Exact Hub repository

Public Hugging Face repository confirmed by the Hub API.

Candidate: useful compatibility or speed evidence. The registry does not offer Run until promotion requirements are met.

Docker configuration

vllm

Container · candidate · docker

Candidate evidence — not a Run contract

Image
verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-ep2-dcp2-v84-language-only@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Digest
sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Port
8000

Observed source tokens

Mechanical split of the source command. Unverified against the engine CLI. The registry does not offer Run for this recipe.

  1. serve
  2. /model
  3. --served-model-name
  4. glm-5.3-flash
  5. --host
  6. 0.0.0.0
  7. --port
  8. 8000
  9. --language-model-only
  10. --tensor-parallel-size
  11. 2
  12. --enable-expert-parallel
  13. --decode-context-parallel-size
  14. 2
  15. --dcp-comm-backend
  16. a2a
  17. --dtype
  18. bfloat16
  19. --load-format
  20. safetensors
  21. --moe-backend
  22. b12x
  23. --attention-backend
  24. B12X_MLA_SPARSE
  25. --kv-cache-dtype
  26. nvfp4_ds_mla
  27. --max-model-len
  28. 98304
  29. --max-num-batched-tokens
  30. 2072
  31. --max-num-seqs
  32. 4
  33. --gpu-memory-utilization
  34. 0.986
  35. --enable-chunked-prefill
  36. --no-enable-prefix-caching
  37. --generation-config
  38. /model
  39. --reasoning-parser
  40. glm45
  41. --tool-call-parser
  42. glm47
  43. --enable-auto-tool-choice
  44. --disable-custom-all-reduce
  45. --speculative-config
  46. {"method":"dflash","model":"/draft","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","kv_cache_dtype":"auto"}
FlagValue
--served-model-nameglm-5.3-flash
--host0.0.0.0
--port8000
--tensor-parallel-size2
--decode-context-parallel-size2
--dcp-comm-backenda2a
--dtypebfloat16
--load-formatsafetensors
--moe-backendb12x
--attention-backendB12X_MLA_SPARSE
--kv-cache-dtypenvfp4_ds_mla
--max-model-len98304
--max-num-batched-tokens2072
--max-num-seqs4
--gpu-memory-utilization0.986
--generation-config/model
--reasoning-parserglm45
--tool-call-parserglm47
--speculative-config{"method":"dflash","model":"/draft","num_speculative_tokens":7,"draft_tensor_parallel_size":2,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","kv_cache_dtype":"auto"}

Environment

VariableValue
B12X_GL53_ROUTE128_HYBRID_TAIL1
B12X_GL53_ROUTE128_WIDE1
KV_FP8_ROPE0
NCCL_IB_DISABLE1
NCCL_P2P_LEVEL4
OMP_NUM_THREADS2
VLLM_B12X_GLM_NOPE_NVFP41
VLLM_ENABLE_PCIE_ALLREDUCE1
VLLM_ENGINE_READY_TIMEOUT_S3600
VLLM_EXL3_PREFILL_BLOCK_M128
VLLM_EXL3_PREFILL_TRELLIS1
VLLM_NVFP4_MLA_DYNAMIC_SCALE0
VLLM_NVFP4_MLA_SCALES_FILE/opt/glm53/calibration/glm53_nvfp4_mla_outer_scales_mtp_power2_v2.json
VLLM_PCIE_ALLREDUCE_BACKENDcpp
VLLM_USE_B12X_DCP_A2A1

Mounts

SourceTarget
${MODEL_ROOT}/GLM-5.3-Flash-tr3-4bpw/model (read-only)
${MODEL_ROOT}/GLM-5.3-Flash-DFlash2/draft (read-only)
${MODEL_ROOT}/cache_glm53_tr3_dflash_v84/cache

Measured speed

ConcurrencyContextPrefillDecodeTTFT msStatusSweep
10—147.1123.9observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2-sweep
132,768—1545,372.6observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2-sweep
165,536—152.610,824.4observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2-sweep
132,3236,225—5,193observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2-sweep
164,5156,083—10,607observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2-sweep

Remaining fields

Identity, launch, related records, and measured speed are shown above. This is the rest of the normalized record.

hardware count
2
hardware id
rtx-pro-6000-blackwell-96gb
id
glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2
model instance id
brandonmusic-glm-5-3-flash-tr3-4bpw--exl3-4bpw
recipe source
0xsero
schema version
local-ai-registry/v1
speed sweep ids
glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp2-sweep
status
candidate

capabilities

chat
Yes
reasoning
Yes
tools
Yes
vision
No

engine

graph mode
full-and-piecewise CUDA graphs; DFlash2 full graph capture
name
vllm
version
0.1.dev20111+g7f1e92bec.d20260827

serving

kv cache tokens
151,052
max concurrency
4
max context tokens
98,304
tensor parallel
2
Provenance & metadata (3)

facts

engine.graph mode · provenance · captured at
2026-08-28T16:57:48Z
engine.graph mode · reason source-graph-mode
engine.graph mode · state known

metadata

acceptance · completion
Yes
acceptance · context 96017 tokens
Yes
acceptance · dflash active
Yes
acceptance · health
Yes
acceptance · tools
Yes
dflash accepted tokens after sweep
4,395
full context resident concurrency
1.537
local power limit w each
275
published benchmark conditions · gpu count
2
published benchmark conditions · gpu power limit w each
600
published benchmark conditions · memory clock offset mhz
6,000
published benchmark conditions · published kv cache tokens
184,619
published benchmark conditions · sustained decode
Yes

provenance

captured at
2026-08-28T16:57:48Z

sources

captured atkindurl
2026-08-28T16:57:48Zmodel-cardhuggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw ↗
2026-08-28T16:57:48Zruntime-sourcegithub.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw ↗