# Limitations

Read these before quoting a number from this rematch.

1. **The citation list is small.** evals/realism/facts.json holds 90 verified authorities. It has no GDPR, DORA or Cal. Civ. Code s.1950.5 entries, which are among the most cited authorities in these tasks. On the 31 Oct v2.6 outputs, 49 of 66 answers would fail the strict check, mostly for real authorities. So the strict cite column mostly measures whether a model writes "Unverified", not whether its law is right. It is kept out of hard fails and scores for that reason. Grounding quality comes from the judges.

2. **Arm wording differs on citations.** Arm A says to mark unsure cites "verify"; arm B says "Unverified". The strict column favours arm B by construction. The lenient column accepts both.

3. **Judges are also contestants.** Pascal-1, Argon and Pascal-1 fallback judge answers from their own families. Mitigations: every answer has at least one judge from another family; per-judge means are shown; a cross-family median drops the same-family judge; pairwise never uses a judge from a family in the pair; judges are blind to model names. Family bias can still survive through style preference, and two of the four contestants are OpenAI models, so Pascal-1 is the same family for half the answers.

4. **Pascal-1 fallback judges a subset.** Pascal-1 fallback scores 5 of 20 answers per cell. Its mean per cell rests on 5 answers and is noisy. Panel medians on the other 15 answers come from two judges.

5. **Sample size.** 20 answers per cell over 14 tasks means 1 or 2 answers per task per cell. Bootstrap CIs on a 0 to 5 scale will likely be about plus or minus 0.2 to 0.3. Differences smaller than that are not evidence. The bootstrap treats answers as independent and ignores clustering by task, so true intervals are somewhat wider. Pairwise comparisons have 6 samples per pair and 6 to 12 votes, so Wilson intervals will be wide.

6. **Single run, single date.** Everything ran on 1 Nov 2026 against whatever provider OpenRouter routed to that day. Model versions behind an id, provider routing, latency and prices can change.

7. **Price change.** Gemini 4 Argon is reported to double its price on 1 Dec 2026. The $/answer figures use 1 Nov prices.

8. **English only, fictional inputs.** All prompts and tasks are in English. Task inputs are fictional fixtures based on customer patterns. They are not real client documents and they omit uploaded files, so document handling at scale is not tested.

9. **Single-turn answers.** Each task is one user message and one answer. Multi-turn flows (prepare and send, revisions), tool use, Vault retrieval and the product router are not exercised. The R4 leak check covers only the strings in its list.

10. **Rubric and tasks come from our own team.** The rubric and the "strong answer" notes inside each task were written by the people who wrote the v2.6 prompt. That can favour arm A's structure. The held-out tasks were never shown to the prompt optimizer, but they were read by the people who tuned v2.6.

11. **TTFT depends on routing and reasoning.** Time to first visible token includes hidden reasoning time and provider queueing. It is a rough guide to user wait, not a product latency measurement.

12. **Budget stops can unbalance cells.** If a phase hits its budget limit, later samples or judge calls are skipped. The n column shows what each cell got.