qwen36-gptq-int4-rtx3090-vllm-tp4

Launch

Exact materialization of this validated launch contract: digest-pinned image, pinned model revision, and the audited arguments. Self-contained — required assets are fetched from this registry and verified against their recorded sha256 before mounting. Also available as local-ai run qwen36-gptq-int4-rtx3090-vllm-tp4.

1. Pulls the exact container image by sha256 digest — the bytes that were validated, not a floating tag.
2. Fetches any required engine assets from this registry and verifies each against its recorded sha256; the audited launch script verifies them again inside the container before use.
3. Downloads the pinned model revision into your Hugging Face cache on first run (reused afterwards).
4. Serves an OpenAI-compatible API on localhost:12434 — point any client at it.

docker run --rm \
  --gpus all \
  --ipc host \
  --shm-size 32g \
  -p 12434:8000 \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/inference-index/vllm:/root/.cache/vllm \
  vllm/vllm-openai@sha256:0a51ea5b4ae2dc5d81890e5173f54203d2a3ae0cfffe51b8fd2afd4391bfd967 \
  --model \
  Twu31/Qwen3.6-35B-A3B-GPTQ-INT4-W4A16-LowLatency \
  --revision \
  d14fd7339b09364ea2aabc542ea20f7dccd97ca1 \
  --served-model-name \
  Qwen3.6-35B \
  --tensor-parallel-size \
  4 \
  --max-model-len \
  131072 \
  --max-num-seqs \
  16 \
  --kv-cache-dtype \
  fp8 \
  --gpu-memory-utilization \
  0.94 \
  --enable-auto-tool-choice \
  --tool-call-parser \
  qwen3_xml \
  --reasoning-parser \
  qwen3

Measured speed