glm-5-3-flash-awq-4bit-rtx-3090-24gb-vllm-tp8

facts

metadata

localmaxxing · backend

cuda

localmaxxing · hardware label

RTX 3090

localmaxxing · notes

AWQ W4A16 routed-expert checkpoint at HF revision ac8c3e52fd4f; local target-only view omits optional NextN draft shards, so speculative decoding is off. Custom SM86 vLLM fork, TP8+EP8, Triton sparse MLA, Marlin MoE, FP8 KV, piecewise CUDA graphs. Two 64-token warmups and six sequential 512-token measurements; temperature 0, seed 260829, ignore_eos=true. Prefix caching is enabled by the engine; each request used a unique cache_salt and every Prometheus cached-token/hit delta was zero. tokSOut is client steady-state; tokSTotal=(prompt+output)/client E2E per current API wording. Temperature-zero hashes: 6 unique/6; all six runs exhausted the 512-token budget in reasoning_content before visible content. PID/config stable=True; restarts unchanged=True; runner incidents=0; benchmark-window service/kernel suspicious lines=0/0. Per-GPU power is median of each run's mean active draw; peak VRAM is summed across all GPUs.

localmaxxing · observed command

/home/thread/venvs/vllm-glm53-sm86-85872952/bin/python /home/thread/venvs/vllm-glm53-sm86-85872952/bin/vllm serve /mnt/nvme/models/wtdcode/GLM-5.3-Flash-AWQ-W4A16-target-only --served-model-name glm-5.3-flash-awq-sm86-test --host 127.0.0.1 --port 18083 --tensor-parallel-size 8 --pipeline-parallel-size 1 --distributed-executor-backend mp --enable-expert-parallel --enable-ep-weight-filter --all2all-backend allgather_reducescatter --disable-custom-all-reduce --language-model-only --load-format safetensors --safetensors-load-strategy lazy --max-parallel-loading-workers 1 --attention-backend TRITON_MLA_SPARSE --dtype bfloat16 --kv-cache-dtype fp8 --max-model-len 204800 --max-num-seqs 1 --max-num-batched-tokens 256 --enable-chunked-prefill --block-size 128 --kv-cache-memory-bytes 1298373120 --kernel-config {"moe_backend":"marlin","enable_flashinfer_autotune":false} --compilation-config {"mode":"VLLM_COMPILE","cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1],"max_cudagraph_capture_size":1} --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47

localmaxxing · run id

cmtebgn9q006clm01eray6kbm

localmaxxing · tokenized · arguments

/home/thread/venvs/vllm-glm53-sm86-85872952/bin/python, /home/thread/venvs/vllm-glm53-sm86-85872952/bin/vllm, serve, /mnt/nvme/models/wtdcode/GLM-5.3-Flash-AWQ-W4A16-target-only, --served-model-name, glm-5.3-flash-awq-sm86-test, --host, 127.0.0.1, --port, 18083, --tensor-parallel-size, 8, --pipeline-parallel-size, 1, --distributed-executor-backend, mp, --enable-expert-parallel, --enable-ep-weight-filter, --all2all-backend, allgather_reducescatter, --disable-custom-all-reduce, --language-model-only, --load-format, safetensors, --safetensors-load-strategy, lazy, --max-parallel-loading-workers, 1, --attention-backend, TRITON_MLA_SPARSE, --dtype, bfloat16, --kv-cache-dtype, fp8, --max-model-len, 204800, --max-num-seqs, 1, --max-num-batched-tokens, 256, --enable-chunked-prefill, --block-size, 128, --kv-cache-memory-bytes, 1298373120, --kernel-config, {moe_backend:marlin,enable_flashinfer_autotune:false}, --compilation-config, {mode:VLLM_COMPILE,cudagraph_mode:PIECEWISE,cudagraph_capture_sizes:[1],max_cudagraph_capture_size:1}, --enable-auto-tool-choice, --tool-call-parser, glm47, --reasoning-parser, glm47

localmaxxing · tokenized · fidelity

faithful

provenance

captured at

2026-08-30T09:10:02Z