Advisor question routing and slots¶
Before the Advisor answers, routing works out whether the question is one of the brain's published
Questions. If it is, routing makes sure the facts that Question's Answer Keys depend on (its
slots) are known. It sits in front of serve(), behind the advisor_question_routing flag. It
runs only for callers that pass route=True.
This page is the design reference. For what an integration sends and receives, see:
Flow¶
ask ──► classify ──┬─ none ───────────────────────────────────► serve() as today
├─ uncertain ─► NeedsInput(choose_question)
└─ matched ──► fill slots ─┬─ nothing required missing ─► serve() on the resolved text
└─ required missing ─────────► NeedsInput(slots)
reply ──► RouteResume(token, question_id | slots | reply) ──┬─ an answer to what was asked ─► fill slots ──► …
└─ a new question ──────────────► classify (fresh)
- Classify. First narrow the published Questions to a shortlist; the catalog matcher
(
app/eval/catalog.py,match()) can do this. Then aQuestionClassifierdecides among them: - JEV is the classifier we intend to use. It makes a Choice over the shortlist plus
no_match, and one Choice per slot over the slot type's values plusnot_stated. It is called through OpenRouter's Decisions API (typesafe/jev-1.14) with the brain's own OpenRouter key. The key comes fromjev.routing_key, whatevertenant_model_choicesays, and the platform key is never used. (OpenRouter reports these callsis_byok: false: the key is the brain's OpenRouter account, not a provider key registered with OpenRouter.) - The
no_matchoption says it covers a general question that names none of the facts a Question's situation turns on. Without that, generic declines ("What notices do we owe when we decline someone?") matched Q-dee-48 (app/eval/question_routing/jev.md, "Generic declines"). - There is also a fallback classifier that needs no JEV.
The result is a RouteDecision. How it is handled depends on its status:
| Status | Meaning | What happens |
|---|---|---|
matched |
Confident it is this Question | Fill its slots |
none |
Confident it is no Question | Answer exactly as today |
uncertain |
Anything else | Ask the person to choose |
The thresholds were set from the live benchmark, not guessed (app/eval/question_routing/jev.md).
A pick below accept (0.6) is uncertain, whether it is a Question or no_match. A value the
ask states needs slot_accept (0.6). A value or "I don't know" read from a reply needs
reply_accept (0.45).
2. Choose (uncertain only). NeedsInput(reason="choose_question") offers the top candidates and
"None of these". If the person picks NONE_OF_THESE (or types "none", "neither", …), the
question is answered unrouted. A typed reply that isn't a pick the router can read goes to the
classifier over the offered Questions only. If it is confidently none of them, the person has
asked something else: see A new question typed in reply.
3. Fill slots. Each slot's value comes from the first of these that has one:
1. what the question states (source question);
2. the org context store (org). Not yet available: the store doesn't exist yet, so this
step never fills a slot today, and every such slot is asked. When it lands it is keyed by
slot type. If a Question has two slots of the same type, org context fills neither, and both
are asked;
3. what the person told the Advisor when asked (told);
4. an optional slot's declared default (default). This is used only after the person was asked
and skipped it or said they don't know. The ask shows the default, and the answer says it
was assumed.
A slot nobody has asked about is unset. A slot the person said they don't know is unknown.
4. Ask. NeedsInput(reason="slots") lists each slot to ask about: its ask_text, whether it's
required, and its allowed values. Optional slots are asked at most once.
5. Answer. Fill the Question's text with the binding using the same function that produces an
Answer Key's question text (app/eval/generalize.py, text.format(...) with display()).
Retrieval and the cache key use that text. The reasoner gets it and the person's own words
(Routed.reasoner_query), so anything more they asked ("…and how long do we keep it?") is still
answered. Their words are in the cache key too, but only when they differ from the text, so an
unrouted answer's key is unchanged. The answer then ends with a "This answer is based on:" note:
which facts it was told, which were assumed from a default, which it hedged over, and (once org
context is available) which came from the organisation's settings.
Rules¶
| Slot | set |
unknown ("I don't know", "unknown", "TBD", …) |
unset |
|---|---|---|---|
| Required | used | stop: say what's needed and why; never guess | ask |
| Optional | used | its default, marked assumed; with no default, a hedged answer covering the values whose Answer Keys differ | ask once (showing any default), then treat as unknown |
Slot filling, not the surface, decides what counts as unknown. A surface passes the person's own
words in RouteResume.slots. A real value that merely contains one of the phrases is not unknown.
A "stop" is a slots ask of the blocked slot. NeedsInput has no blocked shape, and
reason="slots" must ask at least one slot. So a required slot answered "I don't know" is asked
again, on its own, and NeedsInput.message says why the Advisor won't answer without it. A route
takes at most three replies (router.MAX_TURNS). If a required slot is still missing at the last
one, the Advisor replies with a plain message saying what it needs, with no needs_input, and
forgets the route.
A new question typed in reply¶
While a follow-up is pending, a surface passes the person's next plain message on as the reply
(RouteResume.reply). Sometimes it is a new question instead. Routing then forgets the pending route
and routes the typed text fresh, as if no follow-up were pending. The old question is not answered.
- Choosing. The classifier, over the offered Questions only, reads the reply as none of them
(
status="none"). - Slots. All of these hold:
- no slot value came structured with it (
RouteResume.slots); - the extractor read none of the waiting slots from it, neither a value nor "I don't know";
is_unknowndoesn't read it as "I don't know";- it ends in
?, or the classifier, over the whole catalog, matches it to a Question on its own.
Anything else is still the answer. That includes a free-text or date slot's value, which the extractor can't read, and a value that isn't one ("consmer"), which slot filling refuses and says why. Rule 4 is what keeps a free-text value from being misread: a value is rarely phrased as a question, and is never one of the brain's Questions by itself.
When routing fails¶
One rule: answer unrouted, logged. What is answered depends on how far routing got:
- A new ask answers what was asked.
- A reply, once its pending route has loaded, answers the route's original question
(
PendingRoute.asked), never the reply. "Delaware" is not a question. The route is forgotten. This covers every fault after the load, including a Question text that can't be filled (slots.resolve_textraisesSlotErrorfor anystr.formaterror). - A reply whose route is gone (expired, unknown, another actor's) has no original question.
- A typed reply is answered as typed.
- A structured reply (slot values or a pick, with nothing typed) is not a question. On the
WebSocket its
queryis only the SPA's summary ("credit purpose: consumer; …"). The Advisor replies with a plain message, with noneeds_input, saying the follow-up expired and asking the person to ask again. The audit status isroute_expired.
A slot whose type has gone from the registry is asked as free text and never hedged, rather than failing the route.
Pending pointers never outlive the route. The WebSocket's per-conversation pointer and the OpenAI
endpoint's follow-up reference both expire with routing_state.ROUTE_TTL_SECONDS. If a pointer
outlived its route, the person's next, fresh question would be read as a reply to a route that is
gone.
The input guardrail sees what is answered¶
The pre-gate checks the typed query. It also checks a reply's own words (RouteResume.reply) and
every structured slot value (RouteResume.slots) before routing reads them. Once routing has decided
what to answer (the Question's text with its binding, the original question, or a new one), the
pre-gate checks that too, when it differs from what was typed. A block there is the same BLOCK
envelope and pre_block audit row as any other.
How surfaces recognise a reply¶
- WebSocket. A follow-up's token is remembered per conversation. The conversation's next plain
message is the reply. With
advisor_question_routingoff, a leftover pointer is forgotten and the message is routed to a verb as usual. - OpenAI endpoint (stateless). A follow-up's text ends with a short reference,
(ref: 3fa9c2d81b7e), minted per follow-up. Its token is stored under(brain, caller, ref). The next request whose last assistant turn carries that reference is the reply, and the reference is then spent.
The caller is the API key, which every end user of an integrating app shares. A key built from
the follow-up's text, or from the whole history, collided when two of them asked the same thing:
one person's reply resumed the other's route and got the other's answer.
- MCP. The agent passes route_token back with slots, question_id or reply.
Shapes and interfaces¶
- Shapes (
app/models/routing.py): SlotValueRouteCandidateRouteDecisionSlotRequestNeedsInputRouteResumeNONE_OF_THESE
They are mirrored in web/brain-builder/src/lib/api/contract.ts.
- Swappable parts (app/agent/counselor/routing/interfaces.py):
- QuestionClassifier.classify(asked, candidates, registry=...) -> RouteDecision
- SlotExtractor.extract(text, slots, registry=...) -> dict[str, SlotValue]
- CandidateQuestion: what a classifier is shown.
- Envelope. ResponseEnvelope.needs_input is set instead of an answer:
- result then holds the follow-up question as text, so a client that ignores the field still
shows something sensible;
- the field is left out of the serialised envelope when it is unset, so an unrouted answer
serialises as before;
- SCHEMA_VERSION is 1.3.
- serve(route=False, resume=None, route_store=None, require_advisor_key=None). A user-facing
surface passes route=True, and resume with the person's reply. Routes paused on a reply are
kept in route_store (Redis) for routing_state.ROUTE_TTL_SECONDS (30 minutes). It defaults to
cache, and then to the platform Redis. The routing
step itself is in app/agent/counselor/routing/router.py. require_advisor_key defaults to
route; see below.
- Flag. advisor_question_routing, default off, mirrored in the SPA flag registry.
The Advisor needs the brain's OpenRouter key¶
A brain with no OpenRouter key of its own has no Advisor. Every user-facing call refuses, whether
or not the routing flag is on, and whatever the verb. The refusal is a BLOCK envelope with
rule_id="advisor.openrouter_key_required" and a message telling an admin to add the key under the
brain's model providers. The Advisor MCP endpoint returns that envelope as the tool result. The
OpenAI-compatible endpoint turns it into an HTTP 403 OpenAI-style error with
code: openrouter_key_required, since there is no answer to show.
The key check is separate from routing: serve(require_advisor_key=True). It defaults to route,
so a routed call always checks. The verbs that aren't routed pass it explicitly:
- the WebSocket's
retrieve,assessandredline; - the Advisor MCP server's
retrieve; - the operator MCP server's
retrieve,assessandredline.
Evaluation callers pass neither, and are unaffected.
What never changes¶
routenot passed:serve()behaves exactly as before.- Flag off,
route=True, and the brain has its OpenRouter key:serve()answers exactly as before. The key check is the only difference flag-off callers see. - Evaluation is never routed. These callers never pass
route=True: - Question runs;
question_check;advisor_eval;- wiki "deepen".
Scores keep measuring the unrouted path.
- A NeedsInput reply makes no retrieval and no reasoner call, and is never cached.
Names¶
app/eval/routing.py already exists and means something else: which subsystem owns a failing rubric
criterion (DEE-101). Question routing lives under app/agent/counselor/routing/. Its benchmark is
app/eval/question_routing_bench.py.