Local AI Registry

Recipe

glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4

glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4
JSON APIFind compatible recipes

Runtime-validated four-GPU 499,968-token extension of the publisher-qualified GLM-5.3 Flash TR3 EXL3 4bpw profile. Uses TP4/EP4/DCP4, B12X sparse MLA, NVFP4 MLA KV, external DFlash2-7, and full-and-piecewise CUDA graphs. CUDA_DEVICE_MAX_CONNECTIONS=1 and an explicit 1-64 capture set prevent a multi-stream B12X graph-warmup race while retaining the complete C8 DFlash graph shape.

Record

Status
candidate
Source
0xsero
Engine
vllm
Engine version
0.1.dev20111+g7f1e92bec.d20260827
Graph
full-and-piecewise CUDA graphs through 64 target tokens; DFlash2 full graph capture through 8 requests
Accelerators
4
Tensor parallel
4
Context tokens
499,968
Max concurrency
8
KV cache tokens
10,862,941
chat
yes
reasoning
yes
tools
yes
vision
no

Hugging Face model card

Identity

https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw
Repository
brandonmusic/GLM-5.3-Flash-tr3-4bpw
Status
known
Link type
Exact Hub repository

Public Hugging Face repository confirmed by the Hub API.

Candidate: useful compatibility or speed evidence. The registry does not offer Run until promotion requirements are met.

Docker configuration

vllm

Container · candidate · docker

Candidate evidence — not a Run contract

Image
verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-ep2-dcp2-v84-language-only@sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Digest
sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692
Port
8000

Observed source tokens

Mechanical split of the source command. Unverified against the engine CLI. The registry does not offer Run for this recipe.

  1. serve
  2. /model
  3. --served-model-name
  4. glm-5.3-flash
  5. --host
  6. 0.0.0.0
  7. --port
  8. 8000
  9. --language-model-only
  10. --tensor-parallel-size
  11. 4
  12. --enable-expert-parallel
  13. --decode-context-parallel-size
  14. 4
  15. --dcp-comm-backend
  16. a2a
  17. --dtype
  18. bfloat16
  19. --load-format
  20. safetensors
  21. --moe-backend
  22. b12x
  23. --attention-backend
  24. B12X_MLA_SPARSE
  25. --kv-cache-dtype
  26. nvfp4_ds_mla
  27. --max-model-len
  28. 499968
  29. --max-num-batched-tokens
  30. 1024
  31. --max-num-seqs
  32. 8
  33. --compilation-config
  34. {"cudagraph_capture_sizes":[1,2,4,8,16,24,32,40,48,56,64]}
  35. --gpu-memory-utilization
  36. 0.986
  37. --enable-chunked-prefill
  38. --no-enable-prefix-caching
  39. --generation-config
  40. /model
  41. --reasoning-parser
  42. glm45
  43. --tool-call-parser
  44. glm47
  45. --enable-auto-tool-choice
  46. --disable-custom-all-reduce
  47. --speculative-config
  48. {"method":"dflash","model":"/draft","num_speculative_tokens":7,"draft_tensor_parallel_size":4,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","kv_cache_dtype":"auto"}
FlagValue
--served-model-nameglm-5.3-flash
--host0.0.0.0
--port8000
--tensor-parallel-size4
--decode-context-parallel-size4
--dcp-comm-backenda2a
--dtypebfloat16
--load-formatsafetensors
--moe-backendb12x
--attention-backendB12X_MLA_SPARSE
--kv-cache-dtypenvfp4_ds_mla
--max-model-len499968
--max-num-batched-tokens1024
--max-num-seqs8
--compilation-config{"cudagraph_capture_sizes":[1,2,4,8,16,24,32,40,48,56,64]}
--gpu-memory-utilization0.986
--generation-config/model
--reasoning-parserglm45
--tool-call-parserglm47
--speculative-config{"method":"dflash","model":"/draft","num_speculative_tokens":7,"draft_tensor_parallel_size":4,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","attention_backend":"TRITON_ATTN","kv_cache_dtype":"auto"}

Environment

VariableValue
B12X_GL53_ROUTE128_HYBRID_TAIL1
B12X_GL53_ROUTE128_WIDE1
CUDA_DEVICE_MAX_CONNECTIONS1
KV_FP8_ROPE0
NCCL_IB_DISABLE1
NCCL_P2P_LEVEL4
OMP_NUM_THREADS2
VLLM_B12X_GLM_NOPE_NVFP41
VLLM_ENABLE_PCIE_ALLREDUCE1
VLLM_ENGINE_READY_TIMEOUT_S3600
VLLM_EXL3_PREFILL_BLOCK_M64
VLLM_EXL3_PREFILL_TRELLIS1
VLLM_NVFP4_MLA_DYNAMIC_SCALE0
VLLM_NVFP4_MLA_SCALES_FILE/opt/glm53/calibration/glm53_nvfp4_mla_outer_scales_mtp_power2_v2.json
VLLM_PCIE_ALLREDUCE_BACKENDcpp
VLLM_USE_B12X_DCP_A2A1

Mounts

SourceTarget
${MODEL_ROOT}/GLM-5.3-Flash-tr3-4bpw/model (read-only)
${MODEL_ROOT}/GLM-5.3-Flash-DFlash2/draft (read-only)
${MODEL_ROOT}/cache_glm53_tr3_dflash_v84_tp4/cache

Measured speed

ConcurrencyContextPrefillDecodeTTFT msStatusSweep
10—185.371.1observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4-sweep
20—123.8118.9observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4-sweep
40—85.8427.1observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4-sweep
80—45.3301observedglm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4-sweep

Remaining fields

Identity, launch, related records, and measured speed are shown above. This is the rest of the normalized record.

hardware count
4
hardware id
rtx-pro-6000-blackwell-96gb
id
glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4
model instance id
brandonmusic-glm-5-3-flash-tr3-4bpw--exl3-4bpw
recipe source
0xsero
schema version
local-ai-registry/v1
speed sweep ids
glm53-flash-exl3-tr3-4bpw-dflash2-rtxpro6000-vllm-tp4-sweep
status
candidate

capabilities

chat
Yes
reasoning
Yes
tools
Yes
vision
No

engine

graph mode
full-and-piecewise CUDA graphs through 64 target tokens; DFlash2 full graph capture through 8 requests
name
vllm
version
0.1.dev20111+g7f1e92bec.d20260827

serving

kv cache tokens
10,862,941
max concurrency
8
max context tokens
499,968
tensor parallel
4
Provenance & metadata (3)

facts

engine.graph mode · provenance · captured at
2026-08-28T23:44:12Z
engine.graph mode · reason source-graph-mode
engine.graph mode · state known

metadata

acceptance · c8 all streams active
Yes
acceptance · completion
Yes
acceptance · cuda graph capture
Yes
acceptance · dflash active
Yes
acceptance · health
Yes
acceptance · long context prompt tokens
496,045
acceptance · model discovery
Yes
acceptance · omp end to end
Yes
acceptance · tools
Yes
benchmark · c1 decode tok s
185.298
benchmark · c2 aggregate decode tok s
243.819
benchmark · c4 aggregate decode tok s
327.323
benchmark · c8 aggregate decode tok s
345.03
benchmark · method
concurrent streaming finite burst, 512 completion tokens per stream, OpenAI usage accounting
benchmark · profile note
C1-C8 speed sweep was measured on the earlier 98,304-token profile with a 2,048-token scheduler chunk. After promotion to the 499,968-token profile and 1,024-token chunk, C8 was revalidated with all streams active at 229.59 aggregate tok/s over 64 completion tokens per stream.
checkpoint target tensor parallel
2
experimental runtime tensor parallel
4
full context resident concurrency
21.73
kv cache memory gib per gpu
47.33
kv headroom note
The validated 499,968-token profile allocated 10,862,941 KV tokens and reports 21.73 full-context resident requests, exceeding the requested C8 capacity.
local power limit w each
275
long context acceptance · completion tokens
8
long context acceptance · elapsed seconds
383.542
long context acceptance · finish reason
length
long context acceptance · prompt tokens
496,045
long context acceptance · uncached
Yes
runtime state
runtime-tested

provenance

captured at
2026-08-28T23:44:12Z

sources

captured atkindurl
2026-08-28T23:44:12Zmodel-cardhuggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw ↗
2026-08-28T23:44:12Zruntime-acceptancegithub.com/0xSero/local-ai-registry ↗