Skip to content

Questions Framework

The Questions Framework (QF) is how a brain says what a correct answer is, and how Polaris checks that its Advisor gives one.

A team writes down the questions its customers ask. For each one, a domain expert gives the answer they'd stand behind. Polaris turns that answer into a rubric, scores the Advisor against it, and keeps the score over time. When the Advisor answers a question well enough, often enough, a Curator certifies it.

Who it's for

  • Curators and Admins of a brain. They write and approve Questions and Answer Keys, run evals and sign off certification, in the web app's Questions workspace.
  • Outside developers. They do the same things from their own code, with an API key: over the Questions Framework API (REST) or the remote QF MCP endpoint (/v1/mcp/questions). A typical use is an integration that proposes Questions from support tickets, or a CI job that runs evals and reads standing.
  • Agents. They can propose Questions, draft Answer Keys and run evals. They can never approve an Answer Key or sign off a certification. Those are a person's acts.

Every QF feature is off until a platform admin turns it on for the brain. See Access: flags and keys.

Concepts

Question : A question people ask, written once, as a template. It can have slots: placeholders for the facts the answer depends on. "Is a late fee capped in {jurisdiction}?" is one Question with one slot. A Question has no answer of its own. Its Answer Keys do. Each Question has an id (Q-…), a version and a status: draft, published or retired.

Slot : A named fact a Question's answer depends on, such as the state or the product. A slot has a type, is required or optional, and can carry the text the Advisor uses to ask for it. A required slot appears in the Question's text as {name}. An optional slot never does, and can have a default.

Slot type : What a slot's value can be. us_state is an enum of U.S. states, date is an ISO date, corpus_ref is something the corpus names. Polaris ships a few platform types. A brain can add its own, which a second person must approve before anything uses them. See the slot type registry.

Binding : A value for each of a Question's slots, such as {jurisdiction: US-TX}. Its canonical form, the binding key, is slot=value pairs sorted by slot name and joined by &: jurisdiction=US-TX&product=loan. A Question with no slots has one binding, the empty one ("").

Answer Key : The reference answer and rubric for one Question at one binding. It holds the expert's answer and the criteria an answer is scored on. An Answer Key starts as a draft and counts for nothing until a person approves it. Approval locks its content with a hash. To change what counts as correct you write a new version, with a reason.

Rubric and criteria : The rubric is the list of criteria in an Answer Key. Each criterion has a key, a weight and a tier:

  • machine: scored by a pattern, such as "does the answer name the 90-day deadline".
  • judged: scored by a model judge, such as "is the deadline applied correctly". A person can override the judge with a grade.
  • manual: graded only by a person. It never counts toward an automated score.

A criterion can also list the provisions (the rules in the corpus) it rests on. Only a Curator or Admin, person or key, sees a rubric's content. Everyone else sees the Answer Key without it.

Run : One eval: the Advisor answers a Question at each binding a number of times, and Polaris scores every answer against the Answer Key. A live run is the Advisor answering now. An external run scores answers your own system gave. Runs spend the brain's monthly eval budget.

Standing : How well a run scored, from 0 to 1: the median share of the rubric's points its answers earned. A Question's standing over time is a series of points, one per run, and points are only compared when their comparability tuples match.

Comparability tuple : The facts that must match for two scores to be compared: the Answer Key and its version, the Question version, the brain version, the model, the judge model, the seed and the ref (draft, published or offline). A score under a different model or brain version is a different measurement, not a change in the same one.

Certification : A binding is eligible at a brain version when its Answer Key is approved and enough comparable live runs meet the Question's threshold. By default that is at least 5 answered results, a median standing of at least 0.8, every judged criterion graded and no penalty triggered. A Question can raise the threshold, never lower it. Where the Questions workspace is on, an eligible binding is certified once a Curator or Admin signs it off.

The lifecycle

The authoring loop, from an expert's question to a certified answer

The diagram predates one rule change: approving an Answer Key no longer needs a second person. Any Curator or Admin may approve, their own draft included.

  1. Propose a Question. Give its text, its intent (what the asker needs to decide), a topic and its slots. It is written as a draft in the brain's store. It is not served, scored or certified yet. Needs questions:write.
  2. Draft an Answer Key for one binding. Either give the expert's answer, or let the drafter propose a rubric from it (or write the answer from the corpus too). Drafter calls are model calls, checked against the eval budget first. Needs questions:write. A rubric written elsewhere (in the web app's editor, or another brain's) is saved over the draft whole, with a Curator or Admin key.
  3. Approve the Answer Key. A Curator or Admin approves it, in the web app, at the CLI, or with a Curator or Admin key that has answer_key:approve. Approval re-runs the draft's validation gates and refuses an agent identity. The first approval publishes the Question. Anyone with questions:write can instead ask a named Curator to approve it.
  4. Run and evaluate. Start a live run (eval:run), or submit answers your own system gave (eval:submit). Polaris scores each answer, judges the judged criteria, and stores the run with its comparability tuple. People can grade judged criteria to correct the judge (grades:submit).
  5. Read standing. Each run adds a point to the Question's standing. Points under different tuples are kept apart.
  6. Certify. When a binding meets the threshold at the published brain version it becomes eligible. A Curator or Admin signs it off (certification:sign_off). Publishing a new brain version, a new Answer Key version or a changed grade can invalidate a certification, with the reason recorded.

To change a published Question's text or slots, a Curator starts a new Question version (questions:publish). Its bindings are carried over as draft Answer Keys, and the old version stays live until the first of them is approved.

Routing a live question to a Question

The lifecycle above is offline: it measures the Advisor. With advisor_question_routing on, the Advisor also uses published Questions while it answers. It matches a person's question to a published Question, asks for any required slot values the person left out, and then answers the bound question. See Question routing.

Where to go next