base_model:
- huihui-ai/QwQ-32B-abliterated
vllm (pretrained=/root/autodl-tmp/QwQ-32B,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.432 | ± | 0.0314 |
| strict-match | 5 | exact_match | ↑ | 0.744 | ± | 0.0277 |
vllm (pretrained=/root/autodl-tmp/QwQ-32B,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.444 | ± | 0.0222 |
| strict-match | 5 | exact_match | ↑ | 0.716 | ± | 0.0202 |
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.8140 | ± | 0.0125 | |
| - humanities | 2 | none | acc | ↑ | 0.8359 | ± | 0.0251 | |
| - other | 2 | none | acc | ↑ | 0.8103 | ± | 0.0269 | |
| - social sciences | 2 | none | acc | ↑ | 0.8889 | ± | 0.0222 | |
| - stem | 2 | none | acc | ↑ | 0.7544 | ± | 0.0238 |
vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.528 | ± | 0.0316 |
| strict-match | 5 | exact_match | ↑ | 0.740 | ± | 0.0278 |
vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.492 | ± | 0.0224 |
| strict-match | 5 | exact_match | ↑ | 0.742 | ± | 0.0196 |
vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated,add_bos_token=true,max_model_len=4096,dtype=bfloat16,max_num_seqs=3), gen_kwargs: (None), limit: 15.0, num_fewshot: None, batch_size: 1
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.8152 | ± | 0.0126 | |
| - humanities | 2 | none | acc | ↑ | 0.8359 | ± | 0.0253 | |
| - other | 2 | none | acc | ↑ | 0.8000 | ± | 0.0276 | |
| - social sciences | 2 | none | acc | ↑ | 0.8722 | ± | 0.0240 | |
| - stem | 2 | none | acc | ↑ | 0.7754 | ± | 0.0232 |
vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated-awq,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.476 | ± | 0.0316 |
| strict-match | 5 | exact_match | ↑ | 0.752 | ± | 0.0274 |
vllm (pretrained=/root/autodl-tmp/QwQ-32B-abliterated-awq,add_bos_token=true,max_model_len=4096,dtype=bfloat16), gen_kwargs: (None), limit: 500.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.524 | ± | 0.0224 |
| strict-match | 5 | exact_match | ↑ | 0.716 | ± | 0.0202 |
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| mmlu | 2 | none | acc | ↑ | 0.8023 | ± | 0.0130 | |
| - humanities | 2 | none | acc | ↑ | 0.8000 | ± | 0.0266 | |
| - other | 2 | none | acc | ↑ | 0.7949 | ± | 0.0284 | |
| - social sciences | 2 | none | acc | ↑ | 0.8500 | ± | 0.0258 | |
| - stem | 2 | none | acc | ↑ | 0.7789 | ± | 0.0235 |