Sampling parameters to tau2-bench?

#8
by lewtun - opened

Hello Qwen team,

I was wondering if you could share the sampling parameters you used for tau2-bench? By default, the benchmark uses temperature=0 and I wonder if you changed this to the values adopted in the Qwen3 tech report (namely, temperature=0.7, top_p=0.8, top_k=20, and presence_penalty=1.5)?

Also, if you used vllm to generate the trajectories, can you share the command that was used to create the server? Currently I create the server as follows:

vllm serve Qwen/Qwen3-4B-Instruct-2507 \
    --tensor-parallel-size 1 \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --host 0.0.0.0 --port 8000

I am unsure if you use a different --tool-call-parser

Thank you!

lewtun changed discussion title from Sampling parameters to `tau2-bench` to Sampling parameters to tau2-bench?

Sign up or log in to comment