Running the question loop from the CLI¶

How to take a Linear ticket to a measured score. QUESTION_LOOP.md is the method and the reasoning; this is the operating manual.

The work splits in two, and the split is forced rather than chosen:

COLD START     an agent session (Claude Code, has Linear) + a human      once per issue
   ↓
RUN            bb loop run — no Linear, no token                         as often as you like

Approving a fixture is a legal judgement and the one gate that never releases. A cold start therefore already needs a person. Given that, the CLI gains nothing from Linear access it could actually use: it can read a ticket, it can never approve a fixture from one. So it doesn't try.

Part 1 — cold start, in Claude Code¶

You need this once per issue. The output is one approved file: app/eval/fixtures/dee-NN.yaml.

Prompt: draft the fixture¶

Cold-start a question-loop fixture from https://linear.app/deeporb-labs/issue/DEE-60/...

Follow docs/QUESTION_LOOP.md steps 1 and 2:
- question and gold copied VERBATIM from the issue — do not improve the wording
- split machine (does the answer NAME this) from judged (is the claim CORRECT);
  never let one criterion do both
- declare `provisions` per criterion from the issue's own "Rules implicated"
- check which of those the corpus actually holds before you finish, and tell me

Draft it. Do not approve it — show me the criteria and weights and I'll decide.

The agent will come back with the rubric and a coverage check. Read the criteria before approving: they define what counts as correct, and an agent that can change its own scoring will. That is the whole reason this gate exists.

Prompt: sanity-check before spending¶

Take one n=1 run of dee-60 against calliope and show me the answer text and every
criterion. Not as a measurement — as a wiring check.

One answer plus the judged calls. On DEE-58 this caught a judge caller that would otherwise have left 6 of 7 judged criteria ungraded on every run of the baseline. Cheap insurance.

Prompt: is this fixture worth running?¶

Route dee-60 against the current corpus and tell me where its failing points land.
If they are all `retrieval` or `model` the loop cannot act and I should not run it.

This is the one that saves real money. Today dee-37 had 85 points on coverage — the only actionable fixture in the whole suite — while dee-48 had 41 points on retrieval, which is DEE-102 and outside the loop's reach entirely.

Part 2 — run it¶

APP_CONFIG=$(pwd)/env/local/app.env PYTHONHASHSEED=0 uv run bb loop run \
  --fixture dee-37 \
  --brain-id calliope \
  --ref draft \
  --issue https://linear.app/deeporb-labs/issue/DEE-37/...

APP_CONFIG is not optional. Without it the loader falls back to /env/app.env, finds nothing, and every model setting is missing — the command refuses and prints the line to re-run. Export it once if you prefer:

export APP_CONFIG=$(pwd)/env/local/app.env

Logging¶

The loop's own record — every score, per-criterion counts, lost runs, routing, proposals, verdicts, the base gate, and every ingest step — is emitted at INFO under the polaris logger, always. It is not stdout chatter: a run is long, unattended and costs money, and reconstructing one from terminal scrollback is not a plan.

# capture it
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 \
  --brain-id calliope 2>&1 | tee /tmp/dee37.log

# the decision trail afterwards
grep -E "median=|LOST|verdict=|question base|iter |acquired|approved cs" /tmp/dee37.log

-vv adds third-party INFO on top — LightRAG, httpx, retrieval internals. That is noise for reading a run and signal for debugging one: use it when an extraction silently thins or a worker wedges, both of which are invisible at the default level.

PYTHONHASHSEED is required — the command refuses without it and prints the line to re-run. Retrieval is deterministic within a process and varies across them, and the variance lands squarely on which provisions reach the reasoner. Measured on one fixed question and corpus:

PYTHONHASHSEED=0   58 items   35 provision keys
PYTHONHASHSEED=1   62 items   38 provision keys    <- § 1681m arrives only here

Before you run¶

curl -sf localhost:8088/health                   # stores up
pgrep -f librarian_worker || echo "NO WORKER"    # nothing extracts without it

Acquisition stages a document and stops. If the worker is not running, bb loop run refuses the action rather than waiting on a queue nothing drains.

The shapes you'll want¶

# explore — no target, runs until patience, budget or exhaustion stops it
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 --brain-id calliope

# aim at a score
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 --brain-id calliope --target-score 70

# unattended, banking trades that lift the base
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 --brain-id calliope \
  --on-mixed keep --max-fetches 2 --max-actions 4

# stop at every human gate; change sets are left pending for you
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 --brain-id calliope --attended

# noisy fixture — dee-37 has swung 25-50 across identical runs
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 --brain-id calliope --runs 5

# narrow the suite — 9 fixtures x 3 runs is ~54 answers across a run's two scores
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-80 --brain-id themis_lime \
  --suite-fixture dee-57 --suite-fixture dee-70

# everything, including retrieval/extraction/LightRAG internals
APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 uv run bb loop run --fixture dee-37 --brain-id calliope -vv \
  --json-out /tmp/dee37-run.json

Long runs belong in the background — a suite score is ~7–12 minutes and a run is several:

APP_CONFIG=$APP_CONFIG PYTHONHASHSEED=0 nohup uv run bb loop run --fixture dee-37 --brain-id calliope \
  --json-out /tmp/run.json > /tmp/loop.log 2>&1 &
tail -f /tmp/loop.log

Part 3 — reading it¶

answer: together:zai-org/GLM-5.2   brain: calliope   ref: draft   runs: 3
mode  : unattended (draft only)   on-mixed: stop

[baseline] scored in 429s
    dee-37    median=50 spread=25-50 ok=2/3
    ...
[baseline] 2 run(s) LOST of 15: {'dee-37': {"degenerate output: '<answer>'": 1}, ...}

--suite-fixture buys cost with blind spots, and says so. The target is always included — compare_suite returns INCONCLUSIVE without it on both sides. Every score then names what it did not watch, and the report carries unwatched:

[baseline] SAMPLED — 3 fixture(s) scored, 6 NOT watched: dee-37, dee-48, dee-58, dee-59, dee-60,
           dee-62. A regression in those cannot be seen or halt this run.

That line is the whole safety story. A fixture you do not score cannot raise HALT, so a sampled run that ends no_progress is saying "nothing regressed among the ones I looked at" — which is a weaker claim than the same words on a full run, and the report is what lets you tell them apart later. Reach for --suite-runs first: it bounds cost without dropping a question.

Read spread and ok, not median. A fixture that swings 25–50 across identical runs cannot have a change detected by its median. And LOST runs are the first thing to check: a provider failure and a real regression produce the same-looking median.

iter 1: acquire {'keys': ['1026.9(c)(2)(ii)']} -> ['direction'] (30 pts, measured)
  prediction: the corpus holds no text for 1026.9(c)(2)(ii); supplying it should move
              direction. If it does not, the gap was never coverage.

confidence is stated, not implied: measured (a no-model measurement supports it), inferred (routing says so), untested (nobody has evaluated this — the run is the evidence).

  verdict=mixed keep=False halt=False  dee-37 15 -> 70
  question base 0.531 -> 0.655 (▲0.123, tolerance 0.030) [lifted]
    dee-37:+0.550  dee-48:+0.000  dee-57:+0.067  dee-58:+0.000  dee-59:+0.000

Two gates, and neither subsumes the other. compare_suite is per-fixture and categorical; the base gate is the whole question base. A single question collapsing can hide inside a five-question mean, and a slow broad slide is invisible to a categorical per-fixture rule. Both are load-bearing.

The six endings¶

stop	what it means	what you do
`target_reached`	the score was hit	record it, pick the next fixture
`no_progress`	two iterations, no categorical gain	read the register; the remaining points may be out of reach
`budget_exhausted`	a ceiling was hit	raise it deliberately, or accept the answer
`nothing_to_propose`	no failing criterion has an actionable owner	this is a result — the report names which owners hold the points
`collateral_damage`	a non-target fixture regressed categorically	already reverted; look at what the change touched
`corpus_degraded`	the base fell beyond its noise	already reverted, and you were alerted
`mixed_needs_decision`	the target gained, something traded	yours to call. The change is applied (`--on-mixed stop`)

A run that stops early having proved a mechanism does not work is a successful run. The failure to design against is the opposite: a loop that keeps finding something to acquire, because acquisition always succeeds at something.

Part 4 — after the run, in Claude Code¶

The dee-37 run finished — here is /tmp/run.json. Write the register entry per
QUESTION_LOOP.md step 8, whatever the verdict, with the evidence that settled it and
what would reopen it. Then record the portfolio row (kept=False if it was reverted).

Record the negative results. They are the ones nobody else will think to re-derive, and the register is what stops the loop re-attempting a known dead end on its next pass.

Troubleshooting — all of these have actually happened¶

no configuration: APP_CONFIG points at '/env/app.env' — the loader's default is not this repo's config. export APP_CONFIG=$(pwd)/env/local/app.env, or prefix every call. Without it the first symptom is EXTRACTOR_MODEL is not configured, which names a symptom and not the cause; the command now names the cause.

PYTHONHASHSEED is not set — set it. Any value; the same value on both sides of a comparison.

no fixture 'dee-99' — cold-start it first. The command refuses in 0.5s rather than after a 7-minute baseline, because it resolves the fixture before scoring.

worker is NOT running — start it, or acquisition stages documents nothing extracts:

APP_CONFIG=$(pwd)/env/local/app.env nohup uv run python -u -c \
  "import asyncio,sys; from app.librarian_worker import run; asyncio.run(run(sys.argv[1]))" \
  calliope > /tmp/polaris_worker.log 2>&1 &

Use -u. Without it the log is block-buffered and a healthy worker looks dead for 37 minutes.

The worker looks stuck. CPU time is the wrong test — a worker blocked on the provider burns no CPU while working perfectly. Check the log mtime and the HTTP cadence:

tail -3 /tmp/polaris_worker.log; date -r /tmp/polaris_worker.log

A document left status='extracting' with nothing working on it is stale — reset it to extracted and re-queue with bb ingest reprocess --id N --apply.

ThinExtraction: materially thinner than this document has produced before — the guard working. Extraction is not deterministic: the same 55k-char document has yielded 43, then 19, then 63 citations on three runs, because a window can exhaust its output retries and be soft-skipped. Re-extract; do not override unless you have looked.

INCONCLUSIVE — a run failed, so there is no comparison. The loop re-measures up to 3 times on its own. If all three fail, the harness cannot measure at the current failure rate — raise --runs or wait out the provider.

Everything routes to model and nothing is actionable. Suspect the inputs first. A keyed node with no quotable text is not held — the extractor creates one for every provision a document merely cross-references, and 526 of calliope's keyed nodes are such stubs. The loop now routes those to coverage; if you are reading routing yourself, pass textless.