# mmlu-pro-glm53-20260923

Public raw data package for this measurement. Every number on the measurements page citing this window comes from these files.

MMLU-Pro, 12,032 items, K=1: the 2026-09-18–19 suite at 65,536 output tokens, with the 190 truncated items re-run at 163,840 (method reroda_condicional). `mmlu_pro_itens.json` provides, per item, the role (converged, did not converge, infra), whether it was re-run, ground truth, and the letter read by three extractors. Response text is not published; full-file sha256 in the file itself.

Primary figure: MMLU-Pro official extractor (TIGER-AI-Lab/MMLU-Pro, commit f418b116, Apache-2.0), reimplemented in run_mmlu_pro.py (md5 in file). `letra_legado` is the runner's legacy grader, which produced the 72.59% previously reported: it does not match `\boxed{\text{G. …}}` and ignores casing. `letra_boxed` is an ablation (official with `\boxed`) and does not replace the primary figure. `resumo.json` is derived from this package.

Custom 0-shot prompt ('Question / Options / Let's think step by step.'), without the official harness instruction to end with 'the answer is (X)'. Thinking enabled with max effort, T=1.0, top_p 0.95. Unanswered requests are excluded from the denominator; non-finished responses count as errors.

| file | what it is | sha256 (first 16) |
|---|---|---|
| `mmlu_pro_itens.json` | per item: role, ground truth, and letter from each extractor (official, runner legacy grader, official with \boxed); no text | `ff0573635ab03b50` |
| `resumo.json` | means, standard deviations, and window result (K repetitions per arm) | `7b29c4d797892766` |