qwen3-6-27b-nvfp4-dgx-spark-gb10-128gb-vllm-tp1

facts

metadata

localmaxxing · backend

cuda

localmaxxing · batch size

1

localmaxxing · hardware label

GB10 Grace Blackwell

localmaxxing · notes

Caveat: The benchmark measured the streaming decode rate (time between first and last token). On llama-benchy which uses --exact-tg which measures differently (it counts the full request latency divided by output tokens) the numbers were more conservative PP: 772.7 tok/s (±46.4), TG: 18.0 tok/s (±4.1), Peak TG: 29.7 tok/s (±11.2), TTFT: 311.9 ms (±9.4)

localmaxxing · observed command

``` export CUTE_DSL_ARCH="${CUTE_DSL_ARCH:-sm_121a}" # Prevent OpenBLAS/MKL from spawning too many threads (crashes under systemd) export OPENBLAS_NUM_THREADS="${OPENBLAS_NUM_THREADS:-1}" export OMP_NUM_THREADS="${OMP_NUM_THREADS:-1}" export MKL_NUM_THREADS="${MKL_NUM_THREADS:-1}" export RAYON_NUM_THREADS="${RAYON_NUM_THREADS:-1}" export TOKENIZERS_PARALLELISM="${TOKENIZERS_PARALLELISM:-false}" # --- Paths --- VENV_DIR="${VENV_DIR:-./<local-dir>}" # see unsloth recipe (https://unsloth.ai/docs/models/qwen3.6#dgx-spark-with-nvfp4-quants) VLLM_BIN="$VENV_DIR/bin/vllm" MODEL_PATH="${MODEL_PATH:-$HOME/models/Qwen3.6-27B-NVFP4}" SERVED_NAME="${SERVED_NAME:-qwen36-27b}" # --- Speculative decoding --- SPEC_METHOD="${SPEC_METHOD:-mtp}" DFLASH_MODEL="${DFLASH_MODEL:-$HOME/models/Qwen3.6-27B-DFlash}" if [ "$SPEC_METHOD" = "dflash" ]; then SPEC_TOKENS="${SPEC_TOKENS:-15}" SPEC_CONFIG="{\"method\":\"dflash\",\"model\":\"$DFLASH_MODEL\",\"num_speculative_tokens\":${SPEC_TOKENS}}" else SPEC_TOKENS="${SPEC_TOKENS:-2}" SPEC_CONFIG="{\"method\":\"mtp\",\"num_speculative_tokens\":${SPEC_TOKENS}}" fi # --- Thinking mode --- ENABLE_THINKING="${ENABLE_THINKING:-1}" if [ "$ENABLE_THINKING" = "1" ]; then THINK_KWARGS='{"enable_thinking": true}' REASONING_FLAG="--reasoning-parser qwen3" else THINK_KWARGS='{"enable_thinking": false}' REASONING_FLAG="" fi # --- Serving --- MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}" MAX_NUM_SEQS="${MAX_NUM_SEQS:-8}" MAX_BATCH_TOKENS="${MAX_BATCH_TOKENS:-4096}" GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.92}" # --- Optional flags --- EXTRA_FLAGS=() if [ "${LANGUAGE_ONLY:-1}" = "1" ]; then EXTRA_FLAGS+=(--language-model-only) fi if [ "${CHUNKED_PREFILL:-1}" = "1" ] && [ "$MAX_NUM_SEQS" -gt 1 ]; then EXTRA_FLAGS+=(--enable-chunked-prefill) fi # --- Launch --- exec "$VLLM_BIN" serve "$MODEL_PATH" \ --served-model-name "$SERVED_NAME" \ --host 0.0.0.0 --port 8000 \ --tensor-parallel-size 1 \ --gpu-memory-utilization "$GPU_MEM_UTIL" \ --max-model-len "$MAX_MODEL_LEN" \ --max-num-batched-tokens "$MAX_BATCH_TOKENS" \ --max-num-seqs "$MAX_NUM_SEQS" \ --trust-remote-code \ --speculative-config "$SPEC_CONFIG" \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ $REASONING_FLAG \ --default-chat-template-kwargs "$THINK_KWARGS" \ ${OVERRIDE_GENERATION_CONFIG:+--override-generation-config "$OVERRIDE_GENERATION_CONFIG"} \ "${EXTRA_FLAGS[@]}" \ --seed 0 ```

localmaxxing · run id

cmri5obsn016xmj01joh0exn5

localmaxxing · tokenized · fidelity

lossy

provenance

captured at

2026-08-30T09:26:09Z