# Methods

Date of run: 2026-11-01 (America/Los_Angeles). Harness: run_rematch.py in this folder.

## Question

Which default model and which prompt arm should Pascal-1 use? The answer is one configuration: a model plus a system prompt.

## Contestants

All four ids were checked against https://openrouter.ai/api/v1/models on 1 Nov 2026 and are listed as written.

| OpenRouter id | Family | Input $/M | Output $/M |
|---|---|---|---|
| pascal-1-fallback | Anthropic | 4.00 | 20.00 |
| openai/gpt-6-astra | OpenAI | 10.00 | 50.00 |
| pascal-1 | OpenAI | 2.00 | 10.00 |
| google/gemini-4-argon | Google | 2.00 | 10.00 |

Generation settings, same for every model: streaming, reasoning effort high, temperature 0.2, max 20,000 output tokens, up to 5 retries. The model sees the arm's system prompt and only the task's Input section as the user message.

## Prompt arms

| Arm | File | sha256 (file) | sha256 (text as sent) | Size as sent |
|---|---|---|---|---|
| A, v2.6 pack | prompts/v2.6-pack.md | b4d03ccc6b4b69b610408083250ee3ce5433941c46ebfeb67218e4804c902ab7 | same | 16,096 bytes |
| B, light | prompts/light-arm.md | f34497baa5793d632206207eebc4ccccb03b0389e1741efedb39fa336b28db74 | c5c1946cf6f66efeafb30105e991da31bc991fe468da2826d542c10d8bc6fe41 | 8,448 characters |

Arm A is a byte copy of evals/runs/2026-10-31-measured/system-v2.6.md, the prompt used in the 31 Oct measured bake-off (sections 0 to 7 of the v2.6 general chat pack). Arm B is derived from it. The runner strips the HTML comment (the diff note) from light-arm.md before sending. The full line diff is prompts/v2.6-vs-light.diff.

| v2.6 section | Light arm |
|---|---|
| 0. Cold document workflow (Turn 1 and Turn 2 templates, required headers) | Dropped as a template. Kept: quote and change the document, keep defined terms and parties, do not ask for documents you do not need, draft the safe fallback |
| 1. Reply order (recommendation line, Risk line, work product, Assumptions, Sources) | Dropped. The model picks its own structure |
| 1. Rules (user's words, assumptions not questions, who the user is, money shorthand, never send, Associate spec, cite only real authorities, plain English, no em dashes) | Kept, reworded |
| 1b. Known traps | Kept, every item |
| 1c. Bad and good example | Dropped |
| 1d. Regulated counterparty table | Kept as facts in prose (DORA, Delegated Reg 2025/301, PRA SS2/21, SYSC 8, EBA GL); table format dropped |
| 1e. Fixed answer skeleton, question caps | Skeleton dropped. Kept: answer first, lawyer quote choices, "Not reviewed by a lawyer" |
| 2. Router table | Dropped (not used by a single chat answer) |
| 3. Intake caps and chips | Dropped as numbers. Kept: ask only what changes the answer, after the work |
| 4. Sources block format (Confidence per line, max 8 lines, overall line) | Dropped. Replaced by the citation rules below |
| 5. Banned leak list | Kept, plus "Founder summary" and "Founder intake" |
| 6. Small claims and deposit facts | Kept |
| 7. Stall guidance (visible reasoning lines) | Dropped |
| (new) Citations and truth | Added: unsure cite marked "Unverified" on the same line; never show raw [V1] tokens; never invent parties, facts or law |

The added rules are house rules already enforced by evals/realism/check.py, not new policy. Arm A asks for "verify" where arm B asks for "Unverified"; see the citation columns below.

## Tasks

15 tasks from tasks.jsonl (sha256 564429334f083cda63037bb9aec1a488c6776845e26b361240bd65cdddf67f36), task files in evals/tasks/.

| Set | Source | Task ids |
|---|---|---|
| Bake-off (6 common tasks) | evals/model-bakeoff.md | CG1, CG2, CG3, CG4, 01, FG-L1 |
| Held-out (11) | evals/optimizer/config.json, splits.heldout | CG2, CG4, R2, R4, FG-L2, 03, 05, 11, 18, 22, 23 |

CG2 and CG4 are in both. R4 (internal text leaks) has no Input section, so it is not generated; its leak list runs as a deterministic scan over every answer. That leaves 14 generated tasks.

## Sampling

N = 20 answers per model per arm, 160 answers total. Sample i uses task i mod 14, with tasks in weight order (CG1 and CG2 weight 4 first), so the 6 repeats fall on the heaviest tasks and every cell sees the same task schedule.

Why not 30: at list prices, generation alone for N=30 is about $30.67, which leaves under $10 for judging inside a $40 cap. The recommended N=20 design is estimated at $34.30 (generation $20.44, rubric $9.12, pairwise $4.74). Details in COST-ESTIMATE.md.

## Judges and blinding

- Panel: pascal-1, google/gemini-4-argon, pascal-1-fallback. All three are also contestants.
- Pascal-1 and Argon score every answer. Pascal-1 fallback scores samples 0, 4, 8, 12 and 16 of every cell (5 per cell, the same samples everywhere), to save budget.
- Every answer gets at least one judge from another family. Each judge is reported on its own, together with the panel median (median of the judges that scored the answer) and a cross-family median that drops the same-family judge.
- Judges never see the model name. Model and provider names in the answer text (Claude, GPT, Gemini, OpenAI, Anthropic, Grok) are replaced with "[model]" before judging.
- Judge settings: reasoning effort medium, temperature 0, max 6,000 tokens (rubric) or 5,000 (pairwise). A judge reply that is not valid JSON is retried up to 3 times.

## Rubric

Each judge gets evals/rubric.md (sha256 7b3ea75a2994ec40be6f571809bdc3bb084e0f04e822a575bac0d22e9160c1cf), the task file with old "Measured baseline" notes removed, a deterministic citation note, and the answer. It returns integers 0 to 5 for accuracy, grounding, specificity, actionability and polish, plus a list of P0s with quotes. The answer score is the mean of the five; a P0 from that judge caps it at 2.0. The rubric's e2e_wow dimension is not used here.

Cell statistics: mean of per-answer panel medians with a 95% percentile bootstrap CI (2,000 resamples over answers), median of panel medians, per-judge means, cross-family mean. Inter-judge agreement: Krippendorff alpha (interval metric) over answers with two or more judges, and Spearman rank correlation for each judge pair.

## Pairwise

- Pairs: arm A vs arm B for each model (4 pairs), plus every pair among the top 3 cells by rubric mean, minus repeats.
- 6 samples per pair, spread evenly over the 20 (indices 0, 3, 6, 10, 13, 16), with the same task on both sides.
- Judges: up to 2 per comparison, cheapest first, never from a family in the pair. A pair of two OpenAI models gets Argon and Pascal-1 fallback; Pascal-1 fallback vs Argon gets Pascal-1 only.
- Blinding: order randomized per comparison (seeded, 20261101), names stripped as above. The judge returns 1, 2 or tie.
- Win share: ties count half. 95% Wilson interval over judge votes.

## Hard fails (P0)

An answer is a hard fail if any of these hold.
- A judge reports a P0: invented authority, invented party, raw citation token, claimed send, renamed defined term, different question, or material wrong law or maths.
- Deterministic check: a raw [V1] style token, or any R4 leak string ("Completion note", stage tokens like send:0, "no paid work authorised", "FDL sign off required:" and the rest of the list in run_rematch.py).

Em dashes are counted in results.csv but are not a hard fail.

## Citation columns

- Strict cite P0: the answer has at least one citation that does not resolve to a VERIFIED entry in evals/realism/facts.json (sha256 86ec81c8dcee59039a43ed4826cd9b6be48b52fe5574889ef8ceef70c98cd9b2) and is not marked Unverified on its line or under an Unverified heading. Resolution uses resolve_citations() in evals/realism/check.py (sha256 14040ae209e45b600f07d4f85350a2cd51648ce534b0c58fb65f5219e763303e). It is reported as its own column, not inside hard fails, and it does not cap scores, because facts.json is small (see LIMITATIONS.md).
- Lenient cite P0: the same, but a line that says "verify", "verified" or "verification" also counts as marked. This keeps arm A, which is told to write "verify", comparable with arm B.

## Budget control

- Hard cap $40.00 for everything, judges included, set with `--max-usd`. Spend comes from `usage.cost` in each streamed response, or tokens times list price if cost is missing.
- Before each call the runner reserves its worst case: 1.25 x (prompt characters / 3 x input price + max tokens x output price) + $0.002. A call starts only if spend plus all in-flight reservations plus this one stays under the limit. Afterwards the reservation is replaced by the real cost.
- Phase limits keep money for judging: generation may use up to 62% of the cap ($24.80), generation plus rubric up to 86% ($34.40), pairwise up to 100%.
- Failed calls are charged an estimate (prompt plus streamed text). Every attempt is logged in raw/calls.jsonl, and a resumed run counts it.

## Timing

TTFT is measured on the stream from request start. Two values per answer: first token of any kind, including reasoning, and first visible content token. The report shows the median of the second, which is what a user waits for when reasoning is hidden.

## Software and hashes

| File | sha256 |
|---|---|
| prompts/v2.6-pack.md | b4d03ccc6b4b69b610408083250ee3ce5433941c46ebfeb67218e4804c902ab7 |
| prompts/light-arm.md | f34497baa5793d632206207eebc4ccccb03b0389e1741efedb39fa336b28db74 |
| tasks.jsonl | 564429334f083cda63037bb9aec1a488c6776845e26b361240bd65cdddf67f36 |
| evals/rubric.md | 7b3ea75a2994ec40be6f571809bdc3bb084e0f04e822a575bac0d22e9160c1cf |
| evals/realism/facts.json | 86ec81c8dcee59039a43ed4826cd9b6be48b52fe5574889ef8ceef70c98cd9b2 |
| evals/realism/check.py | 14040ae209e45b600f07d4f85350a2cd51648ce534b0c58fb65f5219e763303e |

run_rematch.py gained report-only additions on 1 Nov (agreement statistics and Wilson intervals) after the CI copy may have been taken. Generation and judging logic did not change. make_results.py computes those statistics from raw/ either way. Record the CI commit's sha256 of run_rematch.py in RESULTS.md when the run lands.