Question loop — improving the Advisor's answer to a question, as an agent¶

DEE-101. A procedure for an agent handed a Linear issue that poses a question and states the correct answer: score the Advisor against it, work out which subsystem owns each failure, act only where acting can help, and keep or revert on the rubric.

Written for an agent driving the MCP tools. Companion to LINKAGE_LOOP.md, and the distinction between them is the first thing to get right.

The question loop

Source: question-loop.puml. Red boxes are human gates; each carries what happens to it in Ralph mode. Re-render with java -jar ~/.local/lib/plantuml.jar -tsvg -tpng docs/question-loop.puml.

This is not the linkage loop, and the difference is measured¶

LINKAGE_LOOP.md optimises corpus linkage. This optimises answer quality. The assumption that the first drives the second has been tested twice here and did not hold:

	linkage	the answer
calliope, full linkage loop	68 → 95, joinability 7.7% → 95.4%	verbatim share stayed 8–11%
removing superseded 12 CFR Part 202	health 75 → 72	quoted provisions 1 → 0, and the answer still cited § 202.9

The second row is the one to remember: the corpus no longer contained Part 202 and the answer reached for it anyway. Corpus health is necessary hygiene and not a proxy for answer quality. A loop that raises it and reports the Advisor improved is reporting something it has not measured.

TL;DR¶

 1. question        ← the Linear issue's prompt, verbatim, as a fixture
 2. related docs    ← the issue's own "Rules implicated" list
 3. held?           → per document AND per failing criterion's provisions
 4. acquire         → Scout, for what is missing. costs a fetch. check the register first
 5. extract         → acquisition is NOT ingestion. the worker must be running
 6. escalations     → R0 identity · R1 provision-shape · R2 provenance · R3 refuse
 6b. bulk resolve   → run the rubric across the WHOLE queue, dry. triage what clears
 7. approve         → the corpus moves here, and only here
 8. rebuild + rescore  → machine decides; judged informs, never authorises
 8b. base gate     → the WHOLE question base: lifted / flat / degraded → dropout
 9. threshold met?  → stop. else: is it linkage? → LINKAGE_LOOP.md
10. else            → advise the next step and go to 3

To actually run it: RUNNING_LOOP_CLI.md — the cold-start prompts, the bb loop run invocations, how to read the output, and every failure mode that has actually happened.

Running this unattended? See Ralph mode — which gates release, which do not, and the three things an unattended run needs that an attended one does not.

Running it: cold start once, then the CLI¶

Steps 1–2 need an agent with Linear access and a human. Steps 3–10 do not need either. The split is not a convenience — approving a fixture is a legal judgement and the one gate that never releases, so a cold start already requires a person. Given that, the CLI gains nothing from Linear access it could use: it can read a ticket, it can never approve a fixture from one.

COLD START — an agent session (has Linear), once per issue
  1. take the question verbatim; 2. read "Rules implicated", declare provisions
     -> draft app/eval/fixtures/dee-NN.yaml
     -> A HUMAN APPROVES IT

RUN — bb, no Linear, no token
  PYTHONHASHSEED=0 bb loop run --fixture dee-37 --brain-id calliope --ref draft

The approved fixture is the complete input — question, gold, criteria, weights, provisions.

flag	what it decides
`--ref draft\\|published`	which corpus is scored and acted on
`--on-mixed keep\\|revert\\|stop`	a cross-question trade is a human call; `stop` (default) halts with the change applied so it is judged in place
`--attended`	stop at each gate. Default is unattended: change sets approved to the draft only, R3 ties resolved keep-both
`--target-score`	omit to explore until patience or budget stops the run
`--judge/--no-judge`	without it the judged tier is ungraded, which is not a failure and decides nothing
`-v` / `-vv`	the loop's own steps / plus library INFO
`--issue <url>`	recorded for provenance, never fetched

It refuses PYTHONHASHSEED unset, and it resolves the fixture before the baseline — a suite score is ~7 minutes and 15 model calls, and "that fixture does not exist" is knowable from disk. Both were learned by paying for them.

--on-mixed keep plus the base gate is a coherent autonomous policy: a trade that lifts the base is banked, one that degrades it halts and alerts regardless. --on-mixed revert is what cost a measured +50 on dee-37 — the run backed out a clean acquisition, could not re-acquire what it had just reverted, and stalled.

Steps 3–9 are the cycle. Step 1 and 2 happen once per issue; re-reading the issue on every iteration is how a loop starts re-acquiring documents it already has.

What you are optimising¶

A fixture (app/eval/fixtures/*.yaml) carries the question, the gold answer, and weighted criteria. question_eval runs it n times and reports how often each criterion was satisfied.

median is the MACHINE score only — it is not the fixture score. RunResult.points comes from eval_fixture.score(), the regex scorer; judged criteria are graded separately and never folded into it. So median=38 on a fixture with machine_max=38 and judged_max=62 means "all 38 machine points", not "38 out of 100". The real score is portfolio.standing() — every non-manual criterion that passed a majority of runs, over the full rubric — and on that same measurement dee-59 stood at 93/100, not 38. On 2026-08-23 this field was read as a fixture score across six fixtures and made a baseline look like a collapse; always divide by machine_max + judged_max, or use standing().
Read criteria before median. This fixture has returned 48 and 78 across identical n=5 runs. A median cannot detect a change at that spread; a criterion moving 0/5 → 5/5 can.
Check errors first. Failed runs leave every statistic silently, so a provider outage and a real regression produce the same-looking median. One measurement here lost 3 of 5 runs to ModelHTTPError 500 and a human had to notice.
Pin PYTHONHASHSEED, or before and after are not comparable. Retrieval is deterministic within a process — five identical calls returned byte-identical grounding, Jaccard 1.000 — and varies across them. Measured on DEE-58 with the question, corpus and lens all fixed:

PYTHONHASHSEED=0   58 items   35 provision keys
PYTHONHASHSEED=1   62 items   38 provision keys   <- § 1681m arrives only here
PYTHONHASHSEED=2   58 items   35 provision keys

A baseline and an after-score run as separate processes therefore differ for a reason that has nothing to do with the change under test, and the difference lands exactly where it does most harm: on which provisions reached the reasoner. Set the same seed for every scoring run in a comparison. Findings robust across seeds — § 1002.9 was absent under all three — can be reported without one; a single-seed verdict on a borderline provision cannot. * You may not edit the fixture. It defines what counts as correct. An agent that can change its own scoring will, and every number after that is worthless. Fixture edits are a separate human act.

Three tiers, and only one of them decides¶

tier	scored by	can authorise a keep
machine	regex / code check	yes
judged	an LLM judge that must quote the answer	yes
manual	a person	no

On dee-57 that is 47 machine + 43 judged. A high judged share is information about the question, not a defect in the rubric: almost all of DEE-57 is conditional logic — accept, reject and no-response produce three different duties — and conditionals are what regexes cannot check.

Split mention from correctness rather than writing a clever regex. DEE-57's first draft gave several criteria a semantic describe and a pattern that only checked whether a phrase appeared; one of them matched "rejecting does not cause adverse action" as a pass, because a regex cannot see polarity. Each claim is now either machine (does the answer name this) or judged (is the claim right), and the two never share a criterion.

gold_when must name machine criteria only — gold is computed from the machine parts map, so a judged key there is never satisfied and gold silently never fires.

Why judged counts. It was originally refused on Goodhart grounds and that reasoning was wrong twice over. This loop optimises the corpus and retrieval — it does not write the answer, so it cannot craft phrasing to please a judge. And the machine tier is no more independent of it: a regex is satisfied by stuffing the corpus as readily as a judge is. Independence from the optimiser is what makes a metric safe, not determinism, and on that test the two tiers are equal.

The remaining objection was variance, and it was measured: five independent gradings of one answer flipped 0 of 6 criteria. Given a fixed answer the judge is deterministic — a stabler instrument than the machine tier, whose 23–29 spread comes from the Advisor varying rather than from the scorer.

So a target score can span both tiers. On dee-57 that is 90 points, of which 52 pass reliably today.

What survives, and it is not fixable by stability: the judge and the Advisor are both language models and may share a misconception about the law. A judged pass then means "two models agree", which is not the same as "correct". Nothing automated catches that — hand-grade one GOLD answer on a schedule and compare. If the judge and the human diverge, the judge is what broke, and it will not announce itself.

manual remains human-only and never decides. That is what is left of the guard, and it is deliberately outside the loop.

The loop¶

1. Take the question from the issue, verbatim¶

The issue carries the prompt and the model answer. Both go into a fixture — and the question text must be copied verbatim, not paraphrased. dee-37.yaml carries a note on what paraphrasing one costs; a reworded question is a different question and every number taken against it is incomparable with every number taken before.

You may draft a fixture. You may not approve one. Its criteria and weights are a statement of what counts as a correct answer, which is a legal judgement — and six of DEE-48's fifteen criteria are marked manual: true precisely because no regex settles them honestly. A generated fixture that converts one of those into a regex that appears to pass has made the rubric worse while making the score better.

Declare each criterion's provisions while you are there (DEE-101 R5). It is what lets step 3 route a failure instead of guessing, and it is the same kind of judgement as writing the criterion.

The issue names its own sources. DEE-48 lists five under Rules implicated — the eCFR pages for § 1002.2 and § 1002.9, the Official Interpretations, and the uscode.house.gov pages for 15 U.S.C. § 1681m and § 1681g.

Prefer these over addresses you construct. corpus_acquire builds a URL from a citation key, which works and is why it exists — but the issue author already named the document they meant, and a constructed address for 1002.2(f) resolves to the whole of § 1002.2 whether or not that is what the criterion needs. The named list is evidence about intent; a constructed URL is an inference.

Gap: no MCP tool takes a URL through the acquire pipeline. corpus_acquire constructs its own address from a key; scout_service.verify_url / stage_url_candidate / target_candidate do exist as services but only the last is exposed. Until that is closed, acquire by key and check the identity gate's verdict against the URL the issue named.

3. Baseline, and find out what is missing¶

question_eval(brain_id=…, fixture="dee-48.yaml", ref="draft", runs=5)

ref="draft" scores the corpus as it stands, which makes this a pre-publish gate rather than a post-publish autopsy. Note via="serve" ignores ref — the production pipeline always answers from latest published — so a draft measurement is necessarily via="direct".

Refuses below 3 runs. That is not a formality: at n=1 any comparison you make afterwards is noise wearing a number.

Take one n=1 run first anyway, and read the answer. Not as a measurement — it cannot support a comparison and is not recorded as one — but as a wiring check. A baseline is the most expensive thing in the loop, and every defect it can carry is silent: a judge whose replies never parse, a fixture whose regex never matches, a judge_ask nobody wired. On DEE-58 that single run cost one answer and seven judge calls and caught a judge caller that would have left 6 of 7 judged criteria ungraded on every run of the baseline. Read the answer text too — it is the only place the corpus's gaps appear in plain language, and this one said outright that the context contained no definition of the term the question turned on.

Route each failing criterion to the subsystem that owns it¶

from app.eval import routing
routing.route(fixture, summary, question_check_result, corpus_keys=keys)

owner	what it means	act?
`coverage`	the corpus holds no text for it	yes — `corpus_acquire` (Scout), costs a fetch
`extraction`	text arrived truncated or empty	yes — re-extract the document
`linkage`	text is held as a label, not a rule	yes — curation, see LINKAGE_LOOP.md
`retrieval`	sound text the retriever did not supply	no — report it
`model`	everything was supplied and the answer still failed	no — report it
`undetermined`	nothing has looked at it yet	no — settle it first
`undeclared`	the criterion names no provisions	no — a gap in the fixture

Always pass corpus_keys. question_check is answer-scoped — it reports on provisions the answer named and is silent about the rest. On calliope the answer named 2 provisions and the fixture declares 9, so without corpus presence the entire worklist came back undetermined. Whether the corpus holds a key is a free fact and settles the coverage case outright.

Silence is not exoneration. A provision question_check did not mention has not been cleared; it has not been looked at. Treating that as model claims the reasoner had the text and failed — the most flattering reading of a corpus, reached by not checking.

Settle the silence with the context probe, or the loop stalls on it. question_check is answer-scoped and corpus presence only settles coverage, so everything else lands on undetermined — which no action reaches. On DEE-58 that swallowed the entire worklist: the answer named two provisions, the fixture declares six, and 169 of 173 failing points routed to undetermined. The generator correctly proposed nothing and the run halted NOTHING_TO_PROPOSE, against a retrieval defect sitting in plain sight.

from app.eval import context_probe
got = await context_probe.supplied(brain_id, fixture.question, declared_keys, ref="draft")
supplied = context_probe.as_states(got, corpus_keys)
routing.route(fixture, summary, findings, corpus_keys=keys, supplied=supplied)

Was this provision in the assembled context is a fact about the retriever, costs one retrieval and no model, and settles undetermined in both directions: not supplied is retrieval, supplied-and-still-failing is model. A probe that could only manufacture actionable work would be a way of talking a loop into acting.

Match on identity, never on mentions. The first version scanned body text with retriever._citation_refs — the helper that finds citations a passage points at, which is what the retriever uses to decide what to fetch next. A cross-reference reading "as provided in paragraph (f) of this section" made 1002.2(f) read as supplied when what had arrived was a sentence pointing at it. That flipped 34 of DEE-58's points from retrieval to model. "Mentioned" and "supplied" are opposites here, and the gap between them is itself the shape of a retrieval failure.

4. Check the register, pre-register, then acquire¶

Before spending a run, ask whether this has already been settled.

from app.eval import register
register.blocking("dee-48", "lift the quote gap ration")   # → Entry, or None
register.predictions("dee-48")["C2_labels_inadequate"]     # what was tried for this criterion

app/eval/register/<fixture>.yaml records every attempt: the change, what it was predicted to do, what it actually did, and the evidence that settled it. retry: false means do not attempt this again — and every blocking entry has to say what new evidence would reopen it, or the register is a wall rather than a record.

It is a repo file rather than a table on purpose. These are few, their value is in being read — in review as much as by the loop — and a database row nobody opens is how RETRIEVE_QUOTE_GAP_ALL gets tried a third time.

The register does not refuse for you. blocking returns what should stop a change and why; refusing is yours. A register that silently vetoed changes would be a second, invisible decision rule competing with decision.compare.

Matching is loose, so it will occasionally flag something related but distinct. That is the right direction to be wrong in — the entry it hands back carries the measurement, and you can see immediately whether it applies. Silently missing a known regression is the expensive failure.

Then pre-register.

State which criterion you expect to move and by what mechanism, ideally from a measurement taken with no model in the loop. That is how C2 was fixed: retrieval was measured directly, the prediction was recorded, then tested — and it went 0/5 → 5/5 and held across three sets.

A change that moves a criterion you did not predict is a finding to investigate, not a success to bank. That is how a coincidence becomes a false mechanism.

Act — and refuse where acting cannot help¶

Only coverage, extraction and linkage are yours. retrieval and model are code changes elsewhere.

This is the discipline the loop lives or dies on. Acquisition is the visible, satisfying remedy, and a loop that reaches for it whenever a criterion fails will fetch documents the corpus already holds — a dead end this project has shipped once. A loop that always finds something to acquire has stopped measuring.

5. Extraction — acquisition is not ingestion¶

Three separate gates, and nothing in the middle of them moves the corpus.

corpus_acquire → SourceDoc(pending) → [EXTRACTION] → ChangeSet(pending) → [APPROVAL] → draft corpus

corpus_acquire fetches a document, proves its identity, and targets it for extraction. That is where it stops. The document sits at SourceDoc.status='pending' until something extracts it, and the corpus does not change until a curator approves the resulting change set.

Check the worker is running before you acquire anything. Nothing drains source.ingest on a schedule except app/librarian_worker.py. If it is not running, extraction never happens:

tail -3 /tmp/polaris_worker.log        # dev.sh starts it; --worker-brain picks the brain

dev.sh starts it now and defaults to the env's DEFAULT_BRAIN_ID — which is not necessarily the brain you are working on. Pass --worker-brain <id>.

Verify the documents actually extracted. Acquisition reporting success and extraction never running looks identical from the outside — Scout reports documents acquired, corpus health reports fine, the review queue is quiet, and the material is simply absent:

# pending → not extracted yet;  extracted → look for its ChangeSet
select(SourceDoc).where(SourceDoc.brain_id == …, SourceDoc.status == "pending")

The drain is FIFO across the whole brain and cannot be targeted. bb ingest reprocess --id N selects what to re-queue; the drain then processes the queue from the front. On this machine that backlog reached 147 documents, 122 of them one regulation nobody had noticed was missing, and four newly acquired documents sat behind all of them. Budget for the queue ahead of your work, or drain it first — roughly 30–90s per document.

Stopping the worker strands whatever it was mid-way through — including a clean SIGTERM, not only SIGKILL. Extraction ends in a long, silent resolve-and-commit tail that can run for minutes, and a document interrupted there is left at status='extracting' forever: the finally that would have marked it failed never runs. Observed on a document that had already logged resolved 76/76 locators and still produced no change set.

Reset them before re-running, and re-queue anything whose event was already consumed:

# status='extracting' with no worker alive is always stale — reset to 'pending'

bb ingest reprocess --brain-id <brain> --id <doc> --apply   # if it never comes back

6. Escalations — the rubric: identity, then provenance¶

Extraction lands a change set. If entity resolution could not settle a collision — the incoming document names a provision the corpus already holds, differently — the change set comes back escalated, and approve refuses until every escalation is decided.

get_change_set(change_set_id)     # each escalation carries its precedence verdict

Four rules, in order. Identity first, then provenance.

R0 — identity. Names that do not canonicalise to the same key are not one concept, and there is no collision to resolve. Measured on a live queue: every genuine collision scored exactly 1.000, every false one landed between 0.85 and 0.95 — CFPB v. Townstone Fin. matched Seventh Circuit at 0.98, a case against the court that decided it. Those resolve keep-both. A duplicate node is recoverable and fold_aliases catches it if the two ever share a key; merging distinct concepts is not recoverable and nothing downstream re-checks it.

R1 — a provision-shaped name is a provision. If the name parses as a citation — or as an Official Interpretation (comment 8(a)-5) — then the side typing it as a law source wins. Not a preference between types: the duty a provision imposes is a relation it has, not what it is, and typing § 1002.8(a) as an Obligation breaks citation-following and hierarchy derivation. It also matches what the corpus already does — 459 of 520 provision-shaped names are typed Statute.

R2 — provenance: originated over referred. Otherwise, rank each side's citations by what their text does with the provision:

relation	rank	what the citation's text is
`is` (or unrelated)	3	the provision's own words
`interprets`	2	commentary on it
`about`	1	a reference to it
no quote at all	0	no evidence — skipped, never counted as weak evidence

R3 — refuse. A tie is not broken. Two sides quoting the provision's own text are not distinguishable by any rule here, and picking one would look identical to a correct answer in every metric downstream.

from app.service import escalation_policy
plan = escalation_policy.plan(escalations, proposed_citations=…, existing_snapshot=…,
                              proposed_entities=…)     # ← R1 cannot run without this
plan["resolutions"]   # what the rubric settles — pass straight to review_change_set
plan["undecided"]     # what it does not. these need a person

Pass proposed_entities. R1 compares declared types, and without them every collision falls through to evidence — on a live queue that was the difference between resolving 43 of 55 and resolving 12.

A blanket "statute always wins" is wrong, which is why R1 is gated on the name. Where the name is provision-shaped R1 already covers it; where it is not, a type ranking would retype validation notice — genuinely a Product or an Obligation — because one extractor guessed Statute. A hardcoded type hierarchy is a legal judgement baked into a generic mechanism, and it fails quietly in the cases it is wrong about.

decided: false means the rubric reached R3: no rule settles it. That is the loop working, not the loop stuck — hand those back rather than reaching for a way past them.

6b. Bulk auto-resolve across the queue, dry first¶

Extraction produces many change sets, and deciding them one at a time is how a queue of 23 turns into an afternoon. Run the rubric over all of them before approving any:

for cs in escalated_change_sets:
    out = escalation_policy.plan(
        escalations, proposed_citations=…, existing_snapshot=draft, proposed_entities=…)
    clears = not out["undecided"]        # ← the only triage that matters

Triage on undecided, not on how many resolved. approve refuses a change set with any undecided escalation, so a set that is 9/10 resolved is exactly as blocked as one that is 0/10. Split the queue in two: the ones that clear entirely go to step 7 as a batch, the rest go to a person with the specific collisions named.

Measured on calliope's live queue — 75 escalations across 23 change sets:

auto-resolved   65 / 75   (86%)      R0 keep-both 20 · R1 provision 45
undecided       10                   4 shapes, none provision-shaped
clears entirely 14 / 23 change sets

The 10 left were Regulation F, validation notice (two casings) and independent mortgage banks — none provision-shaped, none separated by evidence. That is the rubric working: they are exactly the collisions a person should read two descriptions and judge.

Spot-check before you approve in bulk, because a resolution applies the whole entity. There is no "fix the type only" option in the resolution vocabulary — a adopts the incoming entity, b keeps the existing one, and the description comes with whichever wins. On this queue all 45 R1 resolutions accepted the incoming side, so 45 existing descriptions would be replaced. Usually right here, since the incoming side is the one that correctly identified them as provisions, but it is a wider blast radius than the escalation card implies. Read a few.

Never widen the batch with accept_escalation_defaults=True. It resolves by position in a list rather than by evidence, and using it to clear the remaining 9 change sets would replace the rubric with an accident of ordering — while reporting the same "23/23 approved" as doing it properly.

7. Approve — the corpus moves here, and only here¶

Approve only the change sets 6b found clear — every escalation resolved, none undecided:

review_change_set(change_set_id, decision="approve",
                  escalation_resolutions=plan["resolutions"], actor=…)

Approve merges into the draft, runs the deterministic Tier-1 repair chain to a fixed point, and rebuilds the draft lens. Until this call the corpus is untouched — which is why the queue row does not clear at acquisition, and why a score taken before approval measures nothing you did.

Approval is a human act. The evidence rule decides collisions within a change set; it does not decide whether third-party content belongs in a legal corpus. Do not self-approve.

Then confirm the lens actually rebuilt, or the next measurement is meaningless:

lens_state(brain_id=…, ref="draft")     # trustworthy must be true

question_check findings taken against a stale lens are not evidence: the text may be perfectly reachable and simply absent from the index that was searched. routing.route refuses outright when findings_trustworthy is false, and that refusal is correct — do not work around it.

8. Re-ask and re-score¶

Same fixture, same runs, same ref, same model. A comparison across a changed n or a changed model is not a comparison — and nothing in compare checks that for you, because a caller who has changed the question is not going to be saved by a validation error.

question_eval(brain_id=…, fixture="dee-48.yaml", ref="draft", runs=5)

from app.eval.decision import compare
compare(before, after)      # → CATEGORICAL_GAIN | CATEGORICAL_LOSS | MIXED | UNPROVEN | INCONCLUSIVE

Run the judge on the judged tier, then read it — do not score with it.

from app.eval import judge
grades = judge.grade(fixture, answer, ask=my_judge_call)   # ask(system, prompt) -> str
grades.satisfied     # {criterion: bool} — only the ones actually graded
grades.ungraded      # the judge could not be read, errored, or cited text that is not there

The judge must quote the answer back, and supports() checks the sentence is really in it. A yes whose quote the answer does not carry is refused, not downgraded to a no — "the judge was wrong" and "the answer failed" are different facts, and recording the second hides a broken judge behind a plausible score.
ungraded is not a failure, and a lot of it means the matcher, not the judge. The first live run refused 3 of 6 correct citations: the Advisor answers in markdown, the judge copies the sentence and not its **bold**, and it quotes on past the boundary. supports() now drops emphasis and anchors on the opening 60 characters — enough that an invented passage cannot match, forgiving enough that a real one is not failed on formatting. If ungraded is high, check a quote against the answer by hand before blaming the judge.
Fails closed. An unreadable verdict is not a pass. That is the DEE-40 lesson citation_verifier already records, where defaulting to yes shipped claims nobody had cleared.
Give the judge room to think, or it will grade nothing. The judge reply is two short lines, so a small max_tokens looks like the obvious economy — and it is a trap when the judge is a reasoning model. Kimi K2.7-Code spends 280–370 tokens on a private trace before emitting a character of content; capped at 400 the response comes back finish_reason: length with content: '', and every criterion reports unreadable verdict. Measured on DEE-58: 6 of 7 criteria ungraded, restored to 7 of 7 by raising the cap to 2000 and changing nothing else. Nothing about this reads as a budget problem from the outside — ungraded looks like a judge that cannot make up its mind. Budget for the trace, not for the answer, and check the raw reply before concluding anything about the judge.
Never judge with the model that wrote the answer. One distribution agreeing with itself is not an independent check.
Nothing is judged unless you pass ask. There is no default caller, so no run can quietly become a judged run.

Pass judge_ask to evaluate to score the judged tier. Without it those criteria come back passed: None — nothing graded them, which is not a failure and decides nothing. Required rather than defaulted, so a judged run is always one a caller asked for: the judge is a second model and its cost and its verdicts both belong to whoever invoked it.

Audit the judge on a schedule. Grade one GOLD answer by hand every so often and compare. If the judge and the human diverge, the judge is what broke, and nothing else in this loop will tell you.

Only CATEGORICAL_GAIN keeps. The rule is asymmetric on purpose: wrongly keeping a regression becomes the baseline everything after it is measured against, while wrongly reverting a real improvement costs one re-run at a higher n.

a criterion crossing ≤⅕ → ≥⅘ is a result
a median shift with no criterion moving is UNPROVEN — revert and record
a trade (something gained, something lost) is MIXED — a human decision, not a rule's
any failed run is INCONCLUSIVE — do not score around it

Then record it. Add an entry to app/eval/register/<fixture>.yaml whatever the verdict — a negative result is worth more than a positive one, because it is the one nobody else will think to re-derive. An entry needs the evidence that settled it and the date; a verdict without its measurement is an opinion the next person cannot check.

8b. The base gate — lift or drop out¶

Applies in both modes. Step 8 decides whether this change helped this question. This decides whether the change cost you the others, and it is the one rule that holds across the project's lifetime rather than one iteration: the question base must lift. A single question's gain paid for out of the rest is not progress, and nothing else in the loop can see it — compare_suite's HALT is per-fixture and categorical, so four questions each slipping a little pass straight through it while the target "wins".

from app.eval import portfolio
from app.eval import fixture as eval_fixture

fixtures = {f.id: f for f in eval_fixture.load_all()}

# once, before the first iteration — the run's opening row
opening = portfolio.standing(before, fixtures)
portfolio.record(opening, at="2026-08-22", note="opening: before <what you are about to do>")

# every iteration, after re-scoring
mv = portfolio.movement(portfolio.standing(before, fixtures),
                        portfolio.standing(after, fixtures))
print(mv.report())

mv.report() is the line to put in front of a human:

question base 0.395 -> 0.474  (▲0.079, tolerance 0.030)  [lifted]
  dee-37 +0.000   dee-48 +0.000   dee-57 +0.067   dee-58 +0.250

Act on the verdict¶

`mv.verdict`	what it means	what you do
`degraded`	the base fell further than its own noise explains	revert, halt `CORPUS_DEGRADED`, alert the operator. Do not start another iteration.
`flat`	inside the noise floor — no evidence either way	continue; `compare_suite` alone decides keep or revert
`lifted`	the base rose beyond the floor	continue; `compare_suite` still decides keep or revert

from app.eval.run import CORPUS_DEGRADED

if d.keep and not mv.degraded:
    before, cur = after, new          # bank it
else:
    undo()
if mv.degraded:
    log(f"*** OPERATOR ALERT: question base {mv.before:.3f} -> {mv.after:.3f} "
        f"(tolerance {mv.tolerance:.3f}). Reverted and stopping. ***")
    state.halt(CORPUS_DEGRADED, mv.report())

The gate is a veto, not a licence. lifted does not authorise a keep on its own — a change can lift the base while trading away the criterion you were aiming at, and compare_suite is what knows that. degraded overrides a keep in the other direction. Wiring it as an approval instead of a veto is how a loop starts banking changes its own rubric rejected.

Alert the operator on degraded, and mean it. This is the one condition where an unattended run has damaged something nobody asked it to touch. collateral_damage says a named question broke; corpus_degraded says the base did, which is worse because no single fixture will show it.

Two comparisons, and they are not interchangeable¶

Per-iteration — movement(standing(before), standing(after)). Gates this change.
Lifetime — portfolio.lifetime(), the first recorded row against the latest. Answers is the corpus getting better at answering questions than when we started, which is the only question the series exists for.

lt = portfolio.lifetime()     # None until there are two rows — one point is not a trend
if lt:
    print(lt.report())

Report the lifetime figure when a run ends, whatever it stopped on. A run that reverted everything still moved the series if an earlier one did, and a run that banked a gain has not helped if the base is below where it started.

Closing the run, and adding questions¶

Record a row at the end of every run, including one that kept nothing — the series is a record of measurements, not of successes. Pass kept=False for a state you reverted, or the series lies in the most damaging direction:

portfolio.record(portfolio.standing(final, fixtures), at="…", kept=True,
                 note="closing: 2 iterations, 0 kept, stopped collateral_damage")

portfolio.record(portfolio.standing(experiment, fixtures), at="…", kept=False,
                 note="co-implication edges — HALT, reverted")

lifetime() skips kept=False rows. The co-implication experiment measured 0.435, was reverted for a categorical loss, and its row sat at the end of the file claiming the corpus stood there — so the lifetime figure read 0.04 below where the corpus actually was, and the next person to look would have concluded the project was going backwards. Keeping the row is right; it is a real measurement of a real state, and the register points at it. Treating it as the endpoint is not.

Adding a question is a lift in itself, and the metric refuses to score it as one. A fixture present on one side only is skipped on both and named in mv.skipped, so the comparison spanning its arrival measures the questions that were already there. From the next row on it counts like any other. Record a row immediately after adding one, with the fixture named in the note, or the first comparison that includes it will silently be a comparison of a different base.

Do not delete or narrow a fixture to make the base look better. That is the fixture-approval gate (step 1) in a different costume, and it is the one gate that never releases.

9. Threshold met? Stop. Otherwise decide which loop you are in¶

Stop when the criteria you targeted pass. Not when the median looks good — the median is not evidence at this spread, and a loop that stops on it will report success it cannot substantiate.

If they do not pass, the question is which loop the remaining work belongs to, and step 3's routing already answered it:

what is left	where it goes
`coverage` — provisions the corpus lacks	back to step 4, acquire more
`linkage` — text held as a label, not a rule	LINKAGE_LOOP.md, then return here
`extraction` — truncated or empty text	re-extract, then return here
`retrieval` / `model`	out of reach. stop and report

Branching to the linkage loop is a real branch, not a fallback. Go there when routing says linkage, not because the score is stuck and linkage is something you know how to do. Corpus health and answer quality have been measured diverging here — 68 → 95 with the answer unmoved — so running the linkage loop hoping it helps this question is precisely the mistake this document opens with.

When you return, the corpus has changed: rebuild, then re-baseline before comparing anything.

10. Advise the next step, and go to 3¶

Steps 1 and 2 do not repeat. Re-reading the issue each pass is how a loop starts re-acquiring documents it already holds — the issue's document list is fixed, and what changes between passes is the corpus, not the question.

What to carry into the next pass:

the criterion you targeted, and whether it moved — a prediction that missed is worth more than one that landed, because it falsifies a mechanism rather than confirming one;
the register entry you just wrote, so the next pass does not re-attempt it;
what you refused, and why — a retrieval or model finding does not become actionable by being seen twice.

Two passes with no categorical move means stop. Not "try something else" — stop and hand back what is left with its owner. A loop that keeps finding things to acquire after the rubric has stopped moving is spending fetches to look busy, and the reason it can do that forever is that acquisition always succeeds at something.

Ralph mode — running unattended¶

An administrative mode for running this loop with no human in it. Everything above still applies; this section says which gates change, what replaces them, and what does not change at all.

The safety property is not "a person looked at it." It is reversibility plus automatic verification, and where both hold a gate is a convention rather than a control. Draft commits are git. Knobs revert in one write. The rubric scores the result. Where either property fails, the gate stays.

The six gates, and what happens to each¶

gate	attended	Ralph mode
fixture approval (step 1)	human	unchanged — still human. See below.
corpus approval (step 7)	human	released, draft only. Publishing stays human.
tied escalations (R3)	refuse, hand back	keep-both.
`retrieval` / `model` routing (step 9)	refuse	act, within a declared knob set.
`manual` criteria	human	excluded from the target score, as always
judge audit	human	out of band. Not a per-iteration gate.

Fixture approval never releases¶

The agent may draft a fixture; it may not approve its own. This is the one place where the Goodhart argument that failed for the judged tier actually holds: the loop optimises the corpus and retrieval, so it cannot shape an answer to please a grader — but it can absolutely shape a rubric to please itself. A fixture is an input to a Ralph run, authored and approved once before the run starts. That is a precondition, not a per-iteration block.

`retrieval` and `model` are not out of reach — they were unwired¶

The routing table refuses them, and that was a decision rather than a limit. Six retrieval knobs already exist, every one of them reversible:

RETRIEVE_CITATION_LOOKUP      RETRIEVE_CLOSE_QUOTE_GAP
RETRIEVE_INJECT_DEFINITIONS   RETRIEVE_QUOTE_GAP_ALL
RETRIEVE_SECTION_EXPANSION    RETRIEVE_GRAPH_EXPAND

RETRIEVE_SECTION_EXPANSION is off by default and has never been evaluated against any rubric.

Declared in app/agent/counselor/retrieval_options.py (DEE-103 G2). All six are per-brain feature flags; the retriever resolves them env → per-brain flag → code default, and untried() gives the action generator what is still worth trying. In particular:

model means agent_settings, never prompt text. Tuning a declared, bounded setting is reversible; rewriting a system prompt is a code change wearing a config costume.
One knob per iteration. Two at once and compare cannot attribute the result, which turns every subsequent decision into a guess.
The register is the memory. A knob combination already settled must not be re-tried, and a loop with no memory of its own negative results will cycle forever finding the same regression. A register entry names the action it settled, so one recording that acquiring a provision did not help does not also veto a retrieval knob that merely names it.
A knob measured at the value it sits on is not offered. measured_at settles a value, not a knob — flipping a proven one is a known regression dressed as exploration, and moving it back after someone else flipped it is exactly what should be proposed.
A failed measurement settles nothing. decision.settles says which verdicts are evidence: every one except INCONCLUSIVE, including UNPROVEN, because "it moved nothing" is a finding. INCONCLUSIVE means a run errored and there is no comparison, so the action is exactly as untested as before. Feed RunState.settled_knobs() to untried(settled=…) rather than tracking it in the driver — the first live run tracked it locally, marked RETRIEVE_SECTION_EXPANSION tried after one of three runs died, and retired the only declared knob with no rubric evidence at all. A loop that reads an outage as a result stops looking exactly where it should keep looking.
Adding a capability ranks above removing one. At equal weight and confidence the tie once fell to alphabetical order, and the loop's opening move became "turn off graph expansion" instead of "turn on the untested knob" — the same alphabetical tie-break already recorded on DEE-48.

What Ralph mode requires that attended mode does not¶

A regression suite — built (DEE-103 G1). compare scores one fixture; suite.compare_suite scores them all and decides once.

from app.eval import suite
before = await suite.score_suite(brain_id=…, model=…, runs=5, judge_ask=…)
...                                        # make one change
after  = await suite.score_suite(brain_id=…, model=…, runs=5, judge_ask=…)
d = suite.compare_suite(before, after, target="dee-57")
d.keep      # only on a clean gain
d.halt      # stop the RUN, not just this change

Three rules, in order:

INCONCLUSIVE — any fixture failed to measure, including one you were not working on. You cannot bank a change while unable to show it broke nothing.
HALT — a non-target fixture regressed categorically. Not "do not keep this" but stop: a loop that has damaged a question it was not working on has lost the thread, and the next iteration compounds it from a worse baseline. Breaking the target is an ordinary revert — you know what did it.
MIXED — the target gained and something else traded. A cross-question trade is a human call.

A fixture measured on only one side is skipped, not inferred. Treating a missing baseline as a zero would manufacture a regression and halt a run because someone added a fixture mid-flight.

And the standing rule above all of them: the question base must lift over its lifetime. compare_suite's HALT is per-fixture and categorical — it caught RETRIEVE_GRAPH_EXPAND=0 costing dee-48 nine median points for nothing on the target. What it cannot see is four questions each slipping a little, none categorically, while the loop reports a win on the one it was aimed at. app/eval/portfolio.py is that check, run every iteration:

from app.eval import portfolio
mv = portfolio.movement(portfolio.standing(before, fixtures),
                        portfolio.standing(after, fixtures))
mv.degraded        # -> revert, halt CORPUS_DEGRADED, alert the operator
portfolio.record(standing, at=…, note=…)   # append to the lifetime series
portfolio.lifetime()                        # first recorded measurement vs the latest

Each question counts once. The mean is over per-fixture shares of their own maximum, not raw points — a 100-point rubric is not twice as important as a 50-point one, and summing raw points lets a large rubric dominate the metric meant to measure it.
A question added mid-run is skipped on both sides, never read as a zero. Otherwise adding a question manufactures a lift, which is exactly backwards.
The bar widens to the measured spread. At n=3 there is no honest significance test, and pretending otherwise is worse than saying so. What exists is each fixture's own run spread — dee-48 has returned 30-47 of 88 across identical runs — and a drop smaller than that explains nothing. DEFAULT_TOLERANCE is 0.03, widened per-measurement by the observed spread.
The series is a repo file (app/eval/portfolio_history.jsonl), for the same reason the register is: is the corpus getting better at answering questions over time is not a question anyone opens a database to ask.

Measured across the DEE-58 run, which is what the series opens with:

question base 0.395 -> 0.474  (▲0.079, tolerance 0.030)  [lifted]
  dee-37 +0.000   dee-48 +0.000   dee-57 +0.067   dee-58 +0.250

Baseline on calliope, every fixture, n=3, PYTHONHASHSEED=0 — the opening row of the series:

dee-37   15/100   spread 15-15      dee-48   38/88   spread 30-47
dee-57   52/90    spread 23-29      dee-58   42/100  spread 25-25

dee-37 is a weak canary at 15/100. It was written against themis_lime and declares no provisions, so it cannot be routed and has little room left to fall — a regression there may not register. Treat it as present-but-thin coverage until it is either re-baselined on this corpus or given a corpus that suits it.

A budget with a hard halt — built (DEE-103 G3). app/eval/run.py holds one RunState across a run: the budget, the iteration log, and the single place a run decides to end.

from app.eval.run import RunState
from app.mcp._autonomy import Budget

state = RunState(target="dee-57", target_score=70,
                 budget=Budget(max_fetches=4, max_actions=10, max_model_calls=60))
stop = state.should_stop(score=current, proposal=next_action)   # None = continue
state.record(action, verdict, kept=…)
state.report()          # what it did, why it stopped, what it spent, what it could not reach

Six endings, all of them good — the run answered its question or proved it could not: target_reached · nothing_to_propose · budget_exhausted · no_progress · collateral_damage · corpus_degraded.

Budget is reused, not restated. Exhaustion is a stop rather than a truncation — a loop that quietly does the first N iterations and reports looks identical to one that finished — and skipped names what it could not reach so the report cannot imply completeness.
Patience counts iterations without a CATEGORICAL_GAIN, not without activity. Activity is not evidence, and a loop that always finds something to do is the failure being guarded: acquisition always succeeds at something, and so does flipping a knob.
collateral_damage is set the moment the suite sees it, not checked next iteration — continuing even one more pass would build on a baseline the run has already made worse. It outranks reaching the target.
A dimension with no ceiling never stops the run. None means the caller declared none, and it must not be read as zero.

An action generator — built (DEE-103 G4). actions.propose turns a routing table into the next thing to try, or says why there is not one.

from app.eval import actions, register
p = actions.propose(routes_by_fixture, target="dee-57",
                    blocked=lambda f, d, prov: register.blocking(f, d, provisions=prov),
                    knobs=declared_knobs)     # knobs are G2; omit and retrieval proposes nothing
p.action, p.targets, p.points, p.prediction, p.confidence

Deterministic, not reasoned. Routing already names each failure's owner and each owner has one mechanical remedy, so this is enumerate → filter → rank → take one. A generator that reasoned about what to try would put an unauditable step at the centre of an unattended loop.

Confidence is stated, not implied: measured (a no-model measurement supports it), inferred (routing says so), untested (nobody has evaluated this — the run is the evidence). Ranking takes points first, then confidence: a big untested swing is worth trying, but the record must not later suggest anyone knew it would work.

Exhaustion is the stop signal. NothingToPropose carries why — which owners still hold failing points out of reach, and what the register vetoed. "The loop failed" and "no action remains" are different reports.

Pass the provisions to the register check. Prose similarity is not enough, and this failed live: a settled entry scored 0.11 against the very action it had already settled and let it straight through. Entries now record the provision keys they settled, and matching on keys is exact.

Resume semantics. If a run dies mid-iteration the corpus is already committed and the register may not be, so the next pass re-derives a result it has already paid for. Write the register entry as the commit point, before anything else that costs money.

Stopping¶

Stop and write a final report when any of these is true:

the target score is reached
two consecutive iterations with no CATEGORICAL_GAIN — the same rule as attended mode
the budget is exhausted
every remaining failing criterion routes to an owner with no untried action
any iteration produces CATEGORICAL_LOSS on a fixture that is not the target — stop immediately and revert; a loop that damages another question is not making progress
the question base degrades (step 8b) — revert, stop, and alert the operator. Distinct from the line above: that one names the question that broke, this one fires when no single question broke categorically and the base fell anyway

A Ralph run that stops early having proved a mechanism does not work is a successful run. The failure mode to design against is the opposite: a loop that keeps finding something to acquire because acquisition always succeeds at something.

Failure modes¶

The score moved and nothing else did¶

At this fixture's spread that is noise. A tie-break change here was first reported as not working, then on a fresh sample looked worth ~7 median points, and was recorded honestly as "suggestive, not established". compare returns UNPROVEN for exactly this shape. Believe it.

Everything routes to `model` and nothing is actionable¶

Suspect the inputs before the conclusion. Two ways this happens, both silent, both reading as a healthy corpus with no work to do:

corpus_keys was not passed, so every unmentioned provision fell through as undetermined;
the question_check payload was read wrong — findings arrive as one flat findings list carrying state, not as separate unquoted/absent lists. Those exist on the result object and do not survive as_dict, where they collapse into counts under summary.

Acquisition succeeded and the score did not move¶

Expected, and not a failure. Acquisition stages a document; extraction and approval are separate gates. Check SourceDoc.status and the review queue before concluding anything.

What this loop cannot fix¶

Measured on calliope, DEE-48: 13 machine points route to retrieval — B1 and B2 both need § 1002.9(a)(2), which the corpus holds with a verbatim quote and the retriever does not supply.

lens_reachability on a freshly rebuilt, provably non-stale lens probed 40 provisions and reached 7. Every unreachable one came back carrying a sample of the real rule text the corpus holds. That is DEE-102, and it is the largest single lever on answer quality here — and entirely outside this loop.

When you reach that boundary, report it and stop. Performing curation work that cannot help is worse than reporting a gap, because it looks like progress.

Reference¶

app/eval/advisor_eval.py — evaluate(), summarize(), the CLI
app/eval/decision.py — the keep/revert rule, and why the median is not evidence
app/eval/suite.py — the regression suite: score every fixture, halt on collateral damage
app/eval/portfolio.py — the question base as one number, and the lifetime series it must lift
app/eval/actions.py — the action generator: what to try next, and what would falsify it
app/agent/counselor/retrieval_options.py — the declared knobs a loop may move, and their cost
app/eval/run.py — the run ledger: budget, patience, and the one place a run decides to stop
app/eval/loop.py + bb loop run — steps 3-10 as a command; steps 1-2 are the cold start
app/eval/acquire.py — an ACQUIRE proposal executed end to end, with a real undo
app/eval/judge_caller.py — the judge the CLI passes; nothing imports it by default
app/eval/routing.py — owners, precedence, and the refusals
app/eval/context_probe.py — did retrieval actually supply this provision; settles undetermined
app/eval/fixture.py — criteria, weights, manual / judged, and provisions
app/eval/judge.py — the judged tier, the quote-back rule, and why it stays out of the score
app/eval/register.py + app/eval/register/*.yaml — what has been tried, and what not to retry
app/librarian_worker.py — the drain that must be running
app/service/question_check.py — read its module docstring before trusting a finding
question-loop.puml — this loop as an activity diagram, gates marked
LINKAGE_LOOP.md — the corpus-side loop this one routes work to