license: cc-by-4.0
language:

• en
  pretty_name: Response Compliance Confounds Prompt Quality — per-item results
  tags:
• prompt-engineering
• llm-as-a-judge
• evaluation
• construct-validity
• verbosity-bias
• ifeval
  size_categories:
• 1K<n<10K
  configs:
• config_name: main_run
  data_files: main_run_500.csv
• config_name: ablation
  data_files: baseline_120.csv
• config_name: benchmark_manifest
  data_files: benchmark_manifest.csv

Response Compliance Confounds Prompt Quality in LLM-as-a-Judge Scoring — per-item results

Per-item numeric results from a study that measures prompt quality and response
compliance as separate constructs, rather than treating a model's response
score as a proxy for the person's prompting skill.

A four-agent evaluator on a locally hosted Qwen 2.5-7B-Instruct judge scores each
prompt as an artefact before any response exists, then scores the resulting
response twice on identical items: once against criteria extracted from that
prompt, and once against a fixed rubric that never sees it.

Code: https://github.com/shahoismael/prompt-quality-vs-response-compliance
Archived: https://doi.org/10.5281/zenodo.22849706

Authors. Shaho Ismael Hassen (0000-0002-6403-7748),
Ahmed Abdulfatah Abdlrazaq (0000-0002-3054-045X) and
Ekhlas Mohammed Noori (0009-0000-8998-3741),
Salahaddin University-Erbil, Erbil, Iraq.

What this is not

No prompt or response text from LMSYS-Chat-1M or WildChat-1M appears here.
The LMSYS agreement prohibits transfer of the dataset to third parties and
WildChat-1M carries flow-down obligations under the AI2 ImpACT licence. Prompts
are identified by SHA-256 hash only. benchmark_manifest.csv lets anyone who
rebuilds the corpus from the original sources verify it is byte-identical to the
one used here.

Files

main_run_500.csv — 500 rows, one per sampled prompt

S_raw, S_cal are empty for 2 items whose every loose judgment was deferred by
the confidence router, so analyses on them run at n = 496.

baseline_120.csv — 120 rows, the ablation subsample

The same items judged by two baselines on byte-identical prompts and responses:
a naive single-pass judge (naive_*) and a chain-of-thought judge (cot_*),
each with score, bias variance, and per-judgment and whole-item latency.
main_run_idx joins back to the main run. Two rows fail the response screen,
leaving 118.

benchmark_manifest.csv — 4,275 rows

idx, source, prompt_sha256, prompt_words for every prompt in the master
benchmark, so a rebuilt corpus can be verified as identical.

Reproducing the headline numbers

import csv, math
rows = list(csv.DictReader(open("main_run_500.csv")))
f = lambda v: (lambda x: None if math.isnan(x) else x)(float(v)) if v.strip() else None

kept = [r for r in rows if (f(r["response_ttr"]) or 0) >= 0.10]
sub  = [r for r in kept if None not in (f(r["S_raw"]), f(r["response_words"]), f(r["Q"]))]

def r2(xs, ys):
    n = len(xs); mx = sum(xs)/n; my = sum(ys)/n
    sxy = sum((a-mx)*(b-my) for a, b in zip(xs, ys))
    return sxy*sxy / (sum((a-mx)**2 for a in xs) * sum((b-my)**2 for b in ys))

L = [f(r["response_words"]) for r in sub]
Q = [f(r["Q"])              for r in sub]
S = [f(r["S_raw"])          for r in sub]
print(len(sub), r2(L, S), r2(Q, S))   # 496  0.2555  0.0076

Response length explains 25.6% of the prompt-derived compliance score;
prompt quality explains 0.76%. Scored against the fixed rubric instead,
length explains 3.5% — so the effect belongs to the extracted criteria, not
to the judge.

The response screen matters

Two of the 500 responses were repetition loops of 24,271 and 33,864 words
(type-token ratio 0.017 and 0.018, against a next-lowest of 0.190). Fitted with
them included, the length regression returns a slope indistinguishable from zero
and R² = 2.5 × 10⁻⁷ — the opposite conclusion from the same data. Ordinary least
squares places no bound on the leverage of a single observation, and generative
pipelines produce such observations routinely. Apply the screen.

Limitations

One judge model, from one family, at one size. The corpus is English. No prompt
here was written under assessment conditions. Bias variance measures a judge's
agreement with itself, never its accuracy.

Licence

These derived results are released CC-BY-4.0. The code is MIT. Content derived
from IFEval remains Apache-2.0. LMSYS-Chat-1M and WildChat-1M must be obtained
from their own maintainers under their own terms.

Citation

@software{hassen_response_compliance_confounds_prompt_quality,
  author  = {Hassen, Shaho Ismael and Abdlrazaq, Ahmed Abdulfatah and
             Noori, Ekhlas Mohammed},
  title   = {Response Compliance Confounds Prompt Quality in LLM-as-a-Judge
             Scoring: evaluator, results and figure code},
  year    = {2026},
  doi     = {10.5281/zenodo.22849706},
  url     = {https://doi.org/10.5281/zenodo.22849706},
  license = {MIT}
}