Questions Framework¶
The Questions Framework (QF) is how a brain says what a correct answer is, and how Polaris checks that its Advisor gives one.
A team writes down the questions its customers ask. For each one, a domain expert gives the answer they'd stand behind. Polaris turns that answer into a rubric, scores the Advisor against it, and keeps the score over time. When the Advisor answers a question well enough, often enough, a Curator certifies it.
Who it's for¶
- Curators and Admins of a brain. They write and approve Questions and Answer Keys, run evals and sign off certification, in the web app's Questions workspace.
- Outside developers. They do the same things from their own code, with an API key: over the
Questions Framework API (REST) or the
remote QF MCP endpoint (
/v1/mcp/questions). A typical use is an integration that proposes Questions from support tickets, or a CI job that runs evals and reads standing. - Agents. They can propose Questions, draft Answer Keys and run evals. They can never approve an Answer Key or sign off a certification. Those are a person's acts.
Every QF feature is off until a platform admin turns it on for the brain. See Access: flags and keys.
Concepts¶
Question
: A question people ask, written once, as a template. It can have slots: placeholders for the
facts the answer depends on. "Is a late fee capped in {jurisdiction}?" is one Question with one
slot. A Question has no answer of its own. Its Answer Keys do. Each Question has an id
(Q-…), a version and a status: draft, published or retired.
Slot
: A named fact a Question's answer depends on, such as the state or the product. A slot has a
type, is required or optional, and can carry the text the Advisor uses to ask for it. A
required slot appears in the Question's text as {name}. An optional slot never does, and can
have a default.
Slot type
: What a slot's value can be. us_state is an enum of U.S. states, date is an ISO date,
corpus_ref is something the corpus names. Polaris ships a few platform types. A brain can add
its own, which a second person must approve before anything uses them. See
the slot type registry.
Binding
: A value for each of a Question's slots, such as {jurisdiction: US-TX}. Its canonical form,
the binding key, is slot=value pairs sorted by slot name and joined by &:
jurisdiction=US-TX&product=loan. A Question with no slots has one binding, the empty one
("").
Answer Key : The reference answer and rubric for one Question at one binding. It holds the expert's answer and the criteria an answer is scored on. An Answer Key starts as a draft and counts for nothing until a person approves it. Approval locks its content with a hash. To change what counts as correct you write a new version, with a reason.
Rubric and criteria : The rubric is the list of criteria in an Answer Key. Each criterion has a key, a weight and a tier:
- machine: scored by a pattern, such as "does the answer name the 90-day deadline".
- judged: scored by a model judge, such as "is the deadline applied correctly". A person can override the judge with a grade.
- manual: graded only by a person. It never counts toward an automated score.
A criterion can also list the provisions (the rules in the corpus) it rests on. Only a Curator or Admin, person or key, sees a rubric's content. Everyone else sees the Answer Key without it.
Run : One eval: the Advisor answers a Question at each binding a number of times, and Polaris scores every answer against the Answer Key. A live run is the Advisor answering now. An external run scores answers your own system gave. Runs spend the brain's monthly eval budget.
Standing : How well a run scored, from 0 to 1: the median share of the rubric's points its answers earned. A Question's standing over time is a series of points, one per run, and points are only compared when their comparability tuples match.
Comparability tuple
: The facts that must match for two scores to be compared: the Answer Key and its version, the
Question version, the brain version, the model, the judge model, the seed and the ref (draft,
published or offline). A score under a different model or brain version is a different
measurement, not a change in the same one.
Certification : A binding is eligible at a brain version when its Answer Key is approved and enough comparable live runs meet the Question's threshold. By default that is at least 5 answered results, a median standing of at least 0.8, every judged criterion graded and no penalty triggered. A Question can raise the threshold, never lower it. Where the Questions workspace is on, an eligible binding is certified once a Curator or Admin signs it off.
The lifecycle¶
The diagram predates one rule change: approving an Answer Key no longer needs a second person. Any Curator or Admin may approve, their own draft included.
- Propose a Question. Give its text, its intent (what the asker needs to decide), a topic and
its slots. It is written as a draft in the brain's store. It is not served, scored or certified
yet. Needs
questions:write. - Draft an Answer Key for one binding. Either give the expert's answer, or let the drafter
propose a rubric from it (or write the answer from the corpus too). Drafter calls are model
calls, checked against the eval budget first. Needs
questions:write. A rubric written elsewhere (in the web app's editor, or another brain's) is saved over the draft whole, with a Curator or Admin key. - Approve the Answer Key. A Curator or Admin approves it, in the web app, at the CLI, or with
a Curator or Admin key that has
answer_key:approve. Approval re-runs the draft's validation gates and refuses an agent identity. The first approval publishes the Question. Anyone withquestions:writecan instead ask a named Curator to approve it. - Run and evaluate. Start a live run (
eval:run), or submit answers your own system gave (eval:submit). Polaris scores each answer, judges the judged criteria, and stores the run with its comparability tuple. People can grade judged criteria to correct the judge (grades:submit). - Read standing. Each run adds a point to the Question's standing. Points under different tuples are kept apart.
- Certify. When a binding meets the threshold at the published brain version it becomes
eligible. A Curator or Admin signs it off (
certification:sign_off). Publishing a new brain version, a new Answer Key version or a changed grade can invalidate a certification, with the reason recorded.
To change a published Question's text or slots, a Curator starts a new Question version
(questions:publish). Its bindings are carried over as draft Answer Keys, and the old version
stays live until the first of them is approved.
Routing a live question to a Question¶
The lifecycle above is offline: it measures the Advisor. With advisor_question_routing on, the
Advisor also uses published Questions while it answers. It matches a person's question to a
published Question, asks for any required slot values the person left out, and then answers the
bound question. See Question routing.
Where to go next¶
- Walkthrough: one Question from proposal to certification, with an API key.
- Questions Framework API: the REST reference.
- MCP: the remote QF MCP endpoint, the Advisor's MCP endpoint and the operator server.
- Schemas: the Question, Answer Key and slot type formats.
- Access: flags and keys: what to turn on, and which key scopes allow what.
- Question routing: how the Advisor uses Questions while it answers.