Questions Framework API¶
The Questions Framework developer API lets your own system work with a brain's Questions: read and match them, propose new ones, draft Answer Keys, run and submit evals, and (with a Curator or Admin key) approve and certify. This page is the reference for every route. For concepts, flags and how to get a key, see Access, flags and scopes.
The same lifecycle is also available to agents over MCP: see Questions Framework over MCP.
OpenAPI schema¶
The published schema is openapi.json (OpenAPI 3.1). It covers every route on
this page and the OpenAI-compatible Advisor. Load it into your API client or code generator. This
page adds what the schema can't say well: who may call, which flags gate a route, and the
redaction rule.
Three routes take a free-form JSON body, so the schema shows no request body for them:
POST /questions/{question_id}/answers, POST /eval-results/{result_id}/grades and
POST /chat/completions. Their bodies are described below and in the
OpenAI-compatible Advisor page.
Base URL¶
Every route is under the brain it acts on:
Locally the gateway runs on http://localhost:8088. The examples on this page use these shell
variables:
export POLARIS=http://localhost:8088 # or https://<host>
export BRAIN=calliope # your brain id
export KEY=sk-polaris-… # your API key
Authentication¶
Send a bearer credential on every request:
Two kinds of credential work:
- An API key (
sk-polaris-…). A key is bound to one brain and carries a list of scopes. Each route below names the scope it needs. A key for another brain is403 brain_forbidden. A key without the scope is403 insufficient_scope. - A signed-in session (a Polaris access token). A person's role decides what they may do: reads need Reader or above, everything else needs Curator or Admin.
A key's actor is who it acts as. A key minted over REST (POST /v1/brains/{id}/api-keys) acts
as the person who minted it; a key minted with bb apikey mint acts as its --actor. Proposals,
drafts, approvals and sign-offs made with a key are recorded as that actor, and the audit row
records the key's id.
A key's role (Reader, Curator or Admin) matters in two places. Curator acts (approve, sign off, publish a version, edit slots) need a Curator or Admin key. Rubric detail is shown only to a Curator or Admin key (see Redaction).
Curator acts by key. Besides the key's role and the act's scope, the key's actor must be an
active Curator or Admin account of this brain, and not an agent identity (agent:, agent/,
bot:, system:, … (agent)). This is checked every time the key acts, not only when it was
minted, so:
- a key minted by a platform operator can't do Curator acts: the operator has no Curator account in the brain;
- a key stops doing Curator acts once its actor is deactivated, or demoted to Reader;
- a key minted with
bb apikey mintacts as--actor, so for Curator acts pass the login email of a real Curator or Admin of the brain. The CLI's default actor (apikey@polaris) has no account and is refused.
A refused actor is 403: approval_refused with detail.code actor_not_a_curator (or
not_a_person) when approving, and actor_not_a_curator (or not_a_person) as the error code
when signing off, starting a version or editing slots.
To get a key, and for the full list of scopes and flags, see Access, flags and scopes.
Feature flags¶
The API is off until a platform admin turns it on for your brain. All flags default to off.
| Flag | What it gates |
|---|---|
question_store_db |
Every route on this page. |
qf_developer_api |
API keys on every route except the five "DEE-147" routes (submit answers, grade, standing, list eval runs, judge agreement), which need only question_store_db. A signed-in person is not affected by this flag. |
question_catalog |
List, match and read Questions, for anyone. |
question_slot_editing |
Start a Question version and edit slots. |
questions_workspace |
Certification sign-off. |
A flag that is off is 403 feature_disabled, never 404, so you can tell it apart from a missing
Question. The message names the flag.
Conventions¶
Pagination¶
List routes take cursor and limit query parameters and return a page:
Pass next_cursor back as cursor to get the next page. next_cursor: null means the last page.
The cursor is opaque: don't build or change one. A cursor this API didn't return (including one
naming an offset past 1,000,000) is 400 invalid. limit defaults to 50 (100 for GET /eval-runs) and is capped at 500.
curl -s "$POLARIS/v1/brains/$BRAIN/questions?limit=20" -H "Authorization: Bearer $KEY"
curl -s "$POLARIS/v1/brains/$BRAIN/questions?limit=20&cursor=eyJvIjoyMH0" -H "Authorization: Bearer $KEY"
Retrying writes safely (Idempotency-Key)¶
Every POST and PUT in the Answer Keys, Evals (runs) and Approvals groups takes an optional
Idempotency-Key header. Send a unique value (a UUID is fine, up to 255 printable characters). If
the request times out, send it again with the same key. You get the first response back instead of
the write happening twice: one proposed Question, one drafted Answer Key, one paid eval run.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/proposals" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: 6f1c2e0a-3b7d-4c39-9a51-0d2f7b1e9c44" \
-d '{"text": "Do we need a license to lend in {jurisdiction}?", "intent": "Decide whether to apply for a lender license before lending.", "topic": "licensing", "slots": ["jurisdiction:us_state"]}'
How it behaves:
- A key is remembered per brain, per caller and per route, for 24 hours.
- The response carries
Idempotent-Replayed: falsethe first time andIdempotent-Replayed: trueon a replay. - Only a success is remembered. A refusal (say
402 budget_exceeded) is not, so a retry asks again. - The same key with a different request (another path, query or body) is
409 idempotency_conflict. Use a new key for a new request. - A retry while the first request is still running is
409 idempotency_in_progress. Wait and retry. - If the server can't store keys right now, a request that sends the header is
503 idempotency_unavailablerather than running without the guarantee. A request without the header is unaffected.
POST /questions/{question_id}/answers doesn't take the header. It has its own retry rule: reuse the
same external_run_id.
Errors¶
Every refusal uses one envelope:
{
"error": {
"code": "feature_disabled",
"message": "'qf_developer_api' (with 'question_store_db') is disabled for this brain",
"detail": null
}
}
Branch on code. It is stable; message is for people and may change. detail carries extra
fields for some codes (for example required_scope, or the refusal's own code for
approval_refused).
| HTTP | code |
Meaning |
|---|---|---|
| 400 | invalid |
The request is wrong: a bad cursor, a missing field, a binding that doesn't fit the Question, an Idempotency-Key that is too long. message says what. |
| 400 | run_cap_exceeded |
More runs per call than the brain allows (see GET /budget). |
| 401 | unauthorized |
No Authorization: Bearer … header, or an invalid access token. |
| 401 | invalid_api_key |
The sk-polaris-… key doesn't exist. |
| 401 | revoked_api_key |
The key was revoked. |
| 402 | budget_exceeded |
The brain's monthly eval budget would be exceeded. Nothing was spent. |
| 402 | max_cost_exceeded |
The run's estimate is over the max_cost_usd you set. Nothing was spent. detail has estimate_usd and max_cost_usd. |
| 403 | insufficient_scope |
The key lacks the route's scope. detail.required_scope names it. |
| 403 | brain_forbidden |
The key or session is for another brain. |
| 403 | role_forbidden |
A Curator act with a Reader key, or a role that may not use the route. |
| 403 | feature_disabled |
A Questions Framework flag is off for this brain. message names it. |
| 403 | forbidden |
Not allowed for this caller: a platform operator approving a tenant's work, a key grading as someone else, an agent identity grading. |
| 403 | actor_not_a_curator, not_a_person |
A Curator act (sign off, start a version, edit slots, save a rubric, approve a slot type) by a key whose actor isn't an active Curator or Admin account of this brain, or is an agent identity. See Authentication. |
| 403 | approval_refused |
An approval or withdrawal refused because of who is asking. detail.code is not_a_person, not_the_approver, self_approval, approver_required, not_the_requester or actor_not_a_curator. |
| 404 | not_found |
No such Question, Answer Key, run, result, certification or eval op. |
| 409 | conflict |
An id you chose is taken: a question_id on a proposal, an answer_key_id on a draft, or a slot type's name. |
| 409 | approval_refused |
Refused because of the request's state. detail.code says why, for example no_draft, already_requested, approver_unknown, not_pending, expired, draft_changed, gate_failed. |
| 409 | citations_missing |
A rubric criterion cites no provision, or one the corpus doesn't hold. detail.citations lists each check. |
| 409 | not_eligible, invalidated, already_signed, not_the_published_version |
A sign-off refused: the certification is no longer eligible, was invalidated, is already signed, or is for an older brain version. |
| 409 | seed_unpinned, no_brain_version |
A live run refused before spending: the server has no fixed PYTHONHASHSEED, or the brain has no corpus at the ref. detail.remedy may say what to do. |
| 409 | idempotency_conflict, idempotency_in_progress |
See Retrying writes safely. |
| 422 | invalid_request |
The body, path or query failed validation (see below). |
| 503 | judge_unavailable |
The judge model is not available. Retry later, or score without the judge. |
| 503 | idempotency_unavailable |
An Idempotency-Key was sent and can't be honoured right now. |
Validation errors¶
A body, path or query parameter that fails validation is 422 invalid_request in the same
envelope. detail.errors lists each problem:
{
"error": {
"code": "invalid_request",
"message": "the request failed validation",
"detail": {
"errors": [
{ "loc": ["body", "slot"], "msg": "Extra inputs are not permitted", "type": "extra_forbidden" }
]
}
}
}
Request bodies refuse fields they don't know, so a misspelt field is a 422, not silently ignored. Responses may gain fields over time: ignore fields you don't know.
What is never returned (redaction)¶
- Answer text is never returned by any route. Eval results carry scores, never the answer.
- Rubric detail is shown only to a caller who may see rubrics: a Curator or Admin person, a
platform admin, or an API key that has
questions:readand a Curator or Admin role. Rubric detail is an Answer Key'srubric, the drafter'scitations, and a result's per-criterion fields (parts,judged_parts,penalties_fired,ungraded,capped_by,human_grades). - Everyone else gets
rubric: null, no per-criterion fields, and a booleancappedin place of the penalty wording.GET /questions/{question_id}says which you got inrubric_visible. - Judge agreement names criteria by key only, never their wording.
Questions¶
Read the brain's published Questions, match an ask to one, and read slots, standing,
certification and coverage. All need questions:read (a key) or Reader and above (a person).
List published Questions¶
GET /v1/brains/{brain_id}/questions
| Scope | questions:read |
| Flags | question_catalog; a key also needs question_store_db and qf_developer_api |
| Query | cursor, limit, topic (one topic card id) |
| Returns | A page of QuestionSummary: question_id, version, status, text, intent, topic, slots |
{
"items": [
{
"question_id": "Q-lender-license",
"version": 1,
"status": "published",
"text": "Do we need a license to lend in {jurisdiction}?",
"intent": "Decide whether to apply for a lender license before lending.",
"topic": "licensing",
"slots": [
{ "name": "jurisdiction", "type": "us_state", "required": true,
"ask_text": "Which state will you lend in?" }
]
}
],
"next_cursor": null
}
A slot's {name} appears in the text only when the slot is required.
Match an ask to a Question¶
POST /v1/brains/{brain_id}/questions/match
Which published Question a person's question is. No model call.
| Scope | questions:read |
| Flags | as for list |
| Body | question (required, the ask as typed), topic_ids (optional, topic card ids to narrow to first) |
| Returns | status (matched, uncertain or none), question_id, candidates (question_id, score, topic), binding, binding_problems, scoped_to |
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/match" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"question": "Do I need a lender license to make loans in Texas?"}'
{
"status": "matched",
"question_id": "Q-lender-license",
"candidates": [{ "question_id": "Q-lender-license", "score": 0.91, "topic": "licensing" }],
"binding": { "jurisdiction": "US-TX" },
"binding_problems": [],
"scoped_to": []
}
binding holds the slot values the ask states. Check binding_problems before you use it. On
uncertain, ask the person which candidate they meant; never bind on uncertain. An empty
question is 422.
Read a Question¶
GET /v1/brains/{brain_id}/questions/{question_id}
| Scope | questions:read |
| Flags | as for list |
| Returns | The summary fields, plus required_provisions (always, by_binding), answer_shape, certified_bindings (binding keys), answer_keys, rubric_visible |
Each Answer Key has answer_key_id, version, is_draft, status, binding, binding_key,
content_sha, drafted_by, approved_by, approved_at and rubric (null unless you may see
rubrics). Errors: 404 not_found.
A binding key is the canonical form of a binding: slot=value pairs sorted by slot and joined
by &, for example jurisdiction=US-TX&product=loan. A Question with no slots has the empty
binding key "".
A Question's slots¶
GET /v1/brains/{brain_id}/questions/{question_id}/slots
| Scope | questions:read |
| Flags | question_store_db; a key also needs qf_developer_api |
| Returns | question_id, published and draft (each version, text, slots, status, change_reason; draft is null with no open draft), bindings (answer_key_id, binding, binding_key, state, question_version) |
Read this before PUT /questions/{question_id}/slots: you pass back the version you read.
Errors: 404 not_found.
Slot types¶
GET /v1/brains/{brain_id}/slot-types
| Scope | questions:read |
| Flags | as for slots |
| Query | include_proposed (default false): also the brain's proposed types |
| Returns | A list of name, kind (parameter or applicability), base (enum, string, corpus_ref, number, date), namespace, description, values (an enum's values), provision_key_prefix, status (active or proposed), origin (platform or the brain id), proposed_by, approved_by |
The slot types a Question's slots may use: the platform's and the brain's own. Use them to build a
binding. A proposed type is listed only with include_proposed, and no slot may use it until it is
approved (Propose a slot type).
A Question's standing over time¶
GET /v1/brains/{brain_id}/questions/{question_id}/standing
| Scope | questions:read |
| Flags | question_store_db only |
| Query | answer_key_id, binding_key, and tuple filters model, brain_version, ref, seed, judge_model, source; end_customer_id |
| Returns | brain_id, question_id, points: [{at, standing, run_ids, tuple}], oldest first |
question_id may also be a standalone Answer Key's id. Each point carries its comparability
tuple (Question and Answer Key version, brain version, model, judge model, seed, ref,
source). Compare only points whose tuples are equal: a change in any field measures something
else. end_customer_id keeps only points whose runs were all submitted for that end customer.
curl -s "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/standing?source=live&ref=published" \
-H "Authorization: Bearer $KEY"
Certified bindings¶
GET /v1/brains/{brain_id}/questions/{question_id}/certified-bindings
| Scope | questions:read |
| Flags | question_store_db; a key also needs qf_developer_api |
| Returns | brain_id, question_id, items: each binding_key, answer_key_id, answer_key_version, brain_version, standing, threshold, evidence_run_ids, computed_at, signed_by, signed_at |
The bindings certified at the brain's published version, with their evidence runs. A brain that
never published has nothing certified. Errors: 404 not_found.
Question coverage¶
GET /v1/brains/{brain_id}/question-coverage
GET /v1/brains/{brain_id}/question-coverage/check?question_id=…&binding=…
GET /v1/brains/{brain_id}/question-coverage/check?answer_key_id=…
| Scope | questions:read |
| Flags | a key needs question_store_db and qf_developer_api; a person needs none |
| Returns | The matrix: brain_id, corpus_empty, summary, rows. A check: one row plus brain_id and corpus_empty |
Whether the published corpus holds the provisions each Question needs at each binding. Each row
has question_id, binding, answer_keys, status (covered, gap or undeclared), held,
required, missing, provisions and unroutable_criteria. No model call.
binding is the binding key, URL-encoded:
curl -s -G "$POLARIS/v1/brains/$BRAIN/question-coverage/check" \
--data-urlencode "question_id=Q-lender-license" \
--data-urlencode "binding=jurisdiction=US-TX" \
-H "Authorization: Bearer $KEY"
Errors: 400 invalid (a binding part that isn't slot=value), 404 not_found.
Answer Keys¶
Propose a Question and draft its Answer Key. With a Curator or Admin key, approve it, start a new
Question version, and edit slots. Every route here takes an Idempotency-Key.
Propose a Question¶
POST /v1/brains/{brain_id}/questions/proposals → 201
| Scope | questions:write |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Body | text (required, up to 4000 characters; a slotted Question names each slot as {slot}), intent (required, one sentence on what the asker needs to decide), topic (required, a topic card id), slots (each name, name:type, or an object {name, type, required, ask_text, default}), question_id (optional, your own id: Q- then lower-case letters, digits and -), required_provisions (optional, {always: [key…], by_binding: {binding_key: [key…]}}), answer_shape (optional, as GET /questions/{question_id} returns it) |
| Returns | question_id, version, is_draft, store_commit, next_step |
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/proposals" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"text": "Do we need a license to lend in {jurisdiction}?",
"intent": "Decide whether to apply for a lender license before lending.",
"topic": "licensing", "slots": ["jurisdiction:us_state"]}'
{
"question_id": "Q-do-we-need-a-license-to-lend-3fa9c2",
"version": 1,
"is_draft": true,
"store_commit": "9c1e…",
"next_step": "draft its Answer Key: POST /questions/{question_id}/answer-keys/drafts; a Curator publishes the Question"
}
Polaris generates the question_id unless you give one. Giving it keeps ids the same when you copy
Questions from one brain to another; it is 409 conflict when the brain already has that id. The
Question is a draft. It is not served, scored or certified until a Curator publishes it.
No model call. Errors: 400 invalid (for example a slot named in the text but not declared, or a
slot type that is only proposed), 409 conflict.
Draft an Answer Key¶
POST /v1/brains/{brain_id}/questions/{question_id}/answer-keys/drafts → 201
| Scope | questions:write |
| Role (person) | Curator or Admin |
| Flags | as for propose |
| Body | binding (a value for each required slot), answer (an expert's reference answer), use_drafter (default false), answer_key_id (optional, your own id) |
| Returns | answer_key_id, version, agent_drafted, store_commit, estimate_usd, rubric, citations |
Drafts the Answer Key for one binding of the Question. You have three options:
answerwithuse_drafter: true: the drafter proposes the rubric from your answer (one model call).use_drafter: truewithoutanswer: the drafter writes the reference answer from the corpus first (two model calls).answerwithuse_drafter: false: a one-criterion skeleton of your answer. Free.
Drafter calls are checked against the brain's monthly budget before the first one, and recorded
against it. estimate_usd is what they were priced at (null when no model was called). rubric
and citations are null unless you may see rubrics.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/answer-keys/drafts" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"binding": {"jurisdiction": "US-TX"},
"answer": "Yes. Texas requires a license under Tex. Fin. Code § 342.051 for…",
"use_drafter": true}'
The draft scores nothing until it is approved. To give it a whole rubric rather than the skeleton,
save one over it with a Curator key (Save a whole rubric). Next,
ask a Curator to approve it (POST /approvals) or approve it yourself with a Curator key
(POST /answer-keys/{id}/approve).
answer_key_id names the Answer Key instead of having Polaris mint one (409 conflict when it is
taken). A draft Question's first Answer Key is the Question's own draft, named as the Question's id
without Q-, so for that one answer_key_id must be that name.
Errors: 400 invalid (no answer and use_drafter: false; a binding that doesn't fit the
Question's slots; a binding that already has an Answer Key), 402 budget_exceeded,
404 not_found, 409 conflict.
Save a whole rubric (Curator key)¶
PUT /v1/brains/{brain_id}/answer-keys/{answer_key_id}/draft
| Scope | questions:write, on a Curator or Admin key |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Body | rubric (required, as GET /questions/{question_id} returns it to a Curator in answer_keys[].rubric), change_reason (needed over an approved Answer Key: <kind> or <kind>: <note>, kind one of law_changed, new_adjudication, scope_refined, error_corrected) |
| Returns | answer_key_id, version, content_sha, validation (ok, errors, citations), store_commit |
Saves a rubric with all its parts as the Answer Key's draft: criteria with their weights, patterns
and cited provisions, verdicts, gold, gold_when and penalties. An open draft is replaced at its
version. Over an approved Answer Key, the save is the next version, which needs change_reason;
the approved version keeps scoring until the new one is approved. The Answer Key's id, version,
binding and Question are the server's; whatever the rubric says about them is ignored.
A rubric that doesn't load is 400 invalid. One that loads but fails a check is saved, and
validation says why approval would refuse it yet: a failing gate in errors, a criterion with a
missing or unresolved provision in citations. No model call.
Only a Curator or Admin key may save a rubric, because only they may read one, and the key's actor
must be an active Curator or Admin account of this brain (403 actor_not_a_curator).
curl -s -X PUT "$POLARIS/v1/brains/$BRAIN/answer-keys/lender-license-tx/draft" \
-H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" -d @rubric.json
Errors: 400 invalid, 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person,
404 not_found.
Propose a slot type¶
POST /v1/brains/{brain_id}/slot-types → 201
| Scope | questions:write |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Body | name (required, lower-case letters, digits and _), base (required: enum, string, corpus_ref, number, date), namespace (required, how org data is segmented, for example product or timing), description (required), values (an enum's values), kind (parameter, the default, or applicability) |
| Returns | The slot type, status: proposed |
Adds one of the brain's own slot types, for slots no platform type fits. It is proposed: no slot
may use it until someone other than its proposer approves it. 409 conflict when the name is taken.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/slot-types" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"name": "credit_purpose", "base": "enum", "namespace": "product",
"description": "Consumer-purpose or business-purpose credit.",
"values": ["consumer", "business"]}'
Approve a slot type (Curator key)¶
POST /v1/brains/{brain_id}/slot-types/{name}/approve
| Scope | slots:write, on a Curator or Admin key |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Body | note (optional, recorded in the store's commit) |
| Returns | The slot type, status: active |
Makes a proposed slot type usable. The key's actor is the approver, must be an active Curator or
Admin account of this brain, and must not be the proposer (403 approval_refused, detail.code:
self_approval). Errors: 403 role_forbidden, 403 actor_not_a_curator, 404 not_found (no such
proposed type).
Approve an Answer Key (Curator key)¶
POST /v1/brains/{brain_id}/answer-keys/{answer_key_id}/approve
| Scope | answer_key:approve, on a Curator or Admin key |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Body | note (optional, recorded in the audit log) |
| Returns | answer_key_id, approved_by, version, commit, api_key_id |
Approves the Answer Key's current draft directly, with no request needed. It runs the same checks as an approval in the web app: every criterion must cite a provision that resolves, and the draft must pass validation. The key's actor is the approver, and the audit row records the key.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/answer-keys/lender-license-tx/approve" \
-H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" -d '{"note": "Checked against § 342.051."}'
An approval is a curator's claim about what counts as a right answer. Approve only a rubric a person has read.
Errors: 403 role_forbidden (a Reader key, whatever its scopes), 403 approval_refused with
detail.code: not_a_person (the key's actor is an agent identity) or actor_not_a_curator (the
key's actor isn't an active Curator or Admin account of this brain, for example a key a platform
operator minted), 403 forbidden (a platform operator's session), 409 approval_refused (for
example no_draft, already_approved, gate_failed),
409 citations_missing,
404 not_found.
Start a new Question version (Curator key)¶
POST /v1/brains/{brain_id}/questions/{question_id}/versions → 201
| Scope | questions:publish, on a Curator or Admin key |
| Role (person) | Curator or Admin |
| Flags | question_store_db and question_slot_editing; a key also needs qf_developer_api |
| Body | text, slots (each name, type, required, ask_text, default), change_reason (required), fill (a value for each added required slot), drop (removed slots whose bound values may go), dry_run |
| Returns | question_id, version, applied, changes, problems, warnings, dropping, store_commit, answer_key_drafts, skipped |
Starts version N+1 of a published Question with new text or slots. Each binding is carried to the
new version as a draft Answer Key (answer_key_drafts; skipped says which weren't and why).
Version N stays live until the first of those drafts is approved. dry_run: true reports the plan
without writing.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/versions" \
-H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"text": "Do we need a license to make {credit_purpose} loans in {jurisdiction}?",
"slots": [{"name": "jurisdiction", "type": "us_state"},
{"name": "credit_purpose", "type": "credit_purpose"}],
"change_reason": "Licensing differs for consumer and business credit.",
"fill": {"credit_purpose": "consumer"}, "dry_run": true}'
Errors: 400 invalid, 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person (the
key's actor), 403 feature_disabled, 404 not_found.
Edit slots (Curator key)¶
PUT /v1/brains/{brain_id}/questions/{question_id}/slots
| Scope | slots:write, on a Curator or Admin key |
| Role (person) | Curator or Admin |
| Flags | as for versions |
| Body | version (the version you read with GET …/slots), text, slots, fill, drop, dry_run |
| Returns | question_id, version, applied, changes, problems, warnings, dropping, store_commit |
Edits the slots of a draft Question version, or only the ask text (ask_text) of a published
one. Anything else on a published Question needs a new version. dry_run: true reports without
writing. Errors: 400 invalid, 403 role_forbidden, 403 actor_not_a_curator /
403 not_a_person (the key's actor), 403 feature_disabled, 404 not_found.
Evals¶
Estimate and run live evals within the brain's budget, submit answers your own system produced, record grades, and read runs.
Estimate a run¶
GET /v1/brains/{brain_id}/questions/{question_id}/runs/estimate
| Scope | eval:run |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Query | runs (answers per Answer Key, default 1), judge (default true), answer_key_id (one binding; else every binding) |
| Returns | estimate_usd, remaining_usd, monthly_usd, allowed, reason |
What POST …/runs would cost, priced exactly as the run is authorized. Free. It never refuses
over budget: allowed: false and reason say the run would be refused.
curl -s "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/runs/estimate?runs=5" \
-H "Authorization: Bearer $KEY"
Start a live eval run¶
POST /v1/brains/{brain_id}/questions/{question_id}/runs → 202
| Scope | eval:run |
| Role (person) | Curator or Admin |
| Flags | as for estimate |
| Body | runs (default 1), judge (default true), ref (draft or published, default draft), answer_key_id (one binding), max_cost_usd (refuse if the estimate is higher) |
| Returns | op_id, status (running or done), estimate_usd, run_ids |
This costs money. The Advisor answers each binding runs times, and Polaris judges, scores and
stores every answer. The budget, the run cap and your max_cost_usd are checked before any
model call. ref: draft scores the corpus as it stands; only ref: published runs can certify the
published version. Without judge, the judged criteria are ungraded and the run can't certify.
A run takes minutes, so the request returns 202 as soon as the run is past every refusal and
spending. Then poll the op.
The eval run flow¶
-
Start the run and keep the
op_id: -
Poll
GET /v1/brains/{brain_id}/eval-ops/{op_id}untilstatusisdoneorerror: -
Read each run with
GET /v1/brains/{brain_id}/eval-runs/{run_id}.
Notes on polling:
GET /eval-ops/{op_id}needseval:run. Only whoever started the run can see it; anyone else gets404. For a key that means the same key: another key is404even if it has the same actor. An op is held in the gateway's memory, so after a gateway restart it is404too: find the run inGET /eval-runsinstead.- If an Advisor or judge call fails part-way, the answers already produced are still stored.
result.codeis thenpartial(orpersist_failed) anderrorsays what went wrong. - A retry with the same
Idempotency-Keyreturns the sameop_idand starts nothing. If the first request was cut off before it answered, a retry is409 idempotency_in_progressuntil that key's pending record expires (15 minutes), never a second run: find the run inGET /eval-runs.
Errors on start: 400 run_cap_exceeded, 402 budget_exceeded, 402 max_cost_exceeded,
404 not_found, 409 seed_unpinned, 409 no_brain_version, 503 judge_unavailable.
Submit answers to score¶
POST /v1/brains/{brain_id}/questions/{question_id}/answers → 201 (or 200 for a duplicate)
| Scope | eval:submit; also eval:run when judge is true |
| Role (person) | Curator or Admin |
| Flags | question_store_db only |
| Returns | duplicate, run_ids, runs, standing, comparability, authorization |
Send answers your own system produced. Polaris scores them against the Question's rubric and
stores them as source: external. question_id may also be a standalone Answer Key's id.
The body is a JSON object:
| Field | Required | Meaning |
|---|---|---|
answers |
yes | Up to 200 of {"answer_key_id": "…", "text": "…"}. |
tuple |
yes | How the answers were produced: {"model", "brain_version", "ref", "seed"}. All four are needed: a score without them can't be compared with anything. ref is draft or published. |
judge |
no | true also grades the judged criteria. That spends the eval budget and needs eval:run. |
external_run_id |
no | Your id for this submission. Resubmitting it returns the stored runs with duplicate: true (HTTP 200) and writes nothing. |
end_customer_id |
no | Which of your customers the answers were for (up to 128 characters). Filter on it later. |
Never send a score: a field named score, points, share, verdict, judged or passed on
the submission or an answer is 400 invalid.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/answers" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-d '{"answers": [{"answer_key_id": "lender-license-tx", "text": "Yes. Under Tex. Fin. Code § 342.051…"}],
"tuple": {"model": "acme:gpt-x", "brain_version": "147676ad", "ref": "published", "seed": "0"},
"external_run_id": "acme-batch-2026-11-07-001"}'
External runs show in standing but never certify. Errors: 400 invalid, 402 budget_exceeded,
404 not_found, 503 judge_unavailable.
Grade a result¶
POST /v1/brains/{brain_id}/eval-results/{result_id}/grades
| Scope | grades:submit |
| Who | A named person. A signed-in Curator or Admin grades as themself. A key grades only as its own actor, and only if the key's role is Curator or Admin and its actor is an active Curator or Admin of this brain (not an agent identity). |
| Flags | question_store_db only |
| Body | criterion (a judged criterion's key), satisfied (true or false), note (optional), grader (optional; if given, it must be the key's own actor) |
| Returns | The result, rescored, with tuple and run_standing |
A person's grade overrides the judge for that criterion on that result, and the run is rescored.
The result_id is a result's id from GET /eval-runs/{run_id}. Only grade a criterion a person
has read the answer for.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/eval-results/4021/grades" \
-H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
-d '{"criterion": "C3_right", "satisfied": true, "note": "Cites the right section."}'
Errors: 400 invalid (satisfied not a boolean, an unknown criterion), 403 forbidden (a key
naming another grader, an agent identity, a Reader key), 404 not_found.
List eval runs¶
GET /v1/brains/{brain_id}/eval-runs
| Scope | questions:read |
| Flags | question_store_db only |
| Query | question_id, answer_key_id, source (live, external, file, reference), end_customer_id, cursor, limit (default 100) |
| Returns | A page of runs, newest first |
Each run has id, its comparability tuple fields (question_id, question_version,
answer_key_id, answer_key_version, model, judge_model, brain_version, ref, seed,
source), binding_key, standing, rubric_approved, provenance (submitted_by, api_key_id,
external_run_id, end_customer_id), content_sha, cost_usd and created_at.
Read an eval run¶
GET /v1/brains/{brain_id}/eval-runs/{run_id}
| Scope | questions:read |
| Flags | question_store_db; a key also needs qf_developer_api |
| Returns | The run, plus results: each id, run_id, run_index, ok, error, verdict, share, full_credit, and capped or the rubric detail |
Note the flags differ from the list: GET /eval-runs is a DEE-147 route and needs only
question_store_db, while GET /eval-runs/{run_id} is part of the developer API, so a key also
needs qf_developer_api (a signed-in person doesn't).
{
"id": 11,
"question_id": "Q-lender-license",
"answer_key_id": "lender-license-tx",
"binding_key": "jurisdiction=US-TX",
"source": "live",
"ref": "published",
"standing": 0.86,
"cost_usd": 0.1,
"results": [
{ "id": 4021, "run_id": 11, "run_index": 0, "ok": true, "verdict": "YES",
"share": 0.86, "full_credit": false, "capped": false }
]
}
Errors: 404 not_found.
Judge agreement¶
GET /v1/brains/{brain_id}/judge-agreement
| Scope | questions:read |
| Flags | question_store_db only |
| Query | judge_model, question_id |
| Returns | brain_id, criteria: each answer_key_id, criterion, judge_model, both, agreed, rate |
How often the judge's call matched a person's grade, per judged criterion. Criteria are named by key, never by their wording.
Approvals¶
Ask a named Curator to approve a draft, withdraw a request, and sign off certifications.
Every POST here takes an Idempotency-Key.
List pending approval requests¶
GET /v1/brains/{brain_id}/approvals
| Scope | questions:write |
| Role (person) | Curator or Admin |
| Flags | question_store_db; a key also needs qf_developer_api |
| Query | status (only pending), cursor, limit |
| Returns | A page of Approval: id, answer_key_id, status, requested_by, approver, note, created_at, expires_at, decided_by, decided_at |
With a key: the pending requests its actor asked for or is named on, oldest first, without the
draft. A signed-in person sees the requests that name them (an Admin sees every one), with the
draft attached for the named approver. Errors: 400 invalid (another status).
Request an approval¶
POST /v1/brains/{brain_id}/approvals
| Scope | questions:write |
| Role (person) | Curator or Admin |
| Flags | as for list (a person also needs questions_workspace) |
| Body | answer_key_id, approver (the login email of a Curator or Admin of this brain) |
| Returns | The pending Approval |
Asks a named person to approve a draft Answer Key. They decide in the web app, or approve directly with their own Curator key. The request is recorded as asked by the key's actor. A request expires after 14 days, and a draft edited after the request can't be approved under it.
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/approvals" \
-H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"answer_key_id": "lender-license-tx", "approver": "curator@acme.example"}'
Errors: 409 citations_missing (a criterion is uncited or cites a provision that doesn't
resolve, where the Questions workspace is on), 409 approval_refused (no_draft,
already_requested, approver_unknown), 422 invalid_request.
Withdraw an approval request¶
POST /v1/brains/{brain_id}/approvals/{pending_id}/withdraw
| Scope | questions:write |
| Role (person) | Curator or Admin (the requester, or an Admin) |
| Flags | as for list (a person also needs questions_workspace) |
| Body | note (optional) |
| Returns | The withdrawn Approval (status: superseded) |
A key may withdraw only a request its own actor made (403 approval_refused,
detail.code: not_the_requester). A decided request is 409 approval_refused
(not_pending).
Deciding a request (approve or reject) is not on the developer API: the named person decides in
the web app. A Curator or Admin key approves directly with
POST /answer-keys/{id}/approve.
Sign off a certification (Curator key)¶
POST /v1/brains/{brain_id}/certifications/{certification_id}/sign-off
| Scope | certification:sign_off, on a Curator or Admin key |
| Role (person) | Curator or Admin |
| Flags | question_store_db and questions_workspace; a key also needs qf_developer_api |
| Body | note (optional) |
| Returns | certification_id, signed_by, signed_at, api_key_id |
Signs off an eligible certification at the published version, so the binding is certified.
Eligibility is judged again first. The key's actor is the signer, and the audit row records the
key. Where questions_workspace is off, a certification is trusted without a sign-off, so this
route is 403 feature_disabled.
Errors: 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person (the key's actor),
403 forbidden (a platform operator's session), 404 not_found, 409 not_eligible, 409 invalidated, 409 already_signed, 409 not_the_published_version.
Budget¶
The eval budget¶
GET /v1/brains/{brain_id}/budget
| Scope | questions:read |
| Flags | question_store_db; a key also needs qf_developer_api |
| Returns | brain_id, monthly_usd, max_runs_per_call, spent_usd_this_month, remaining_usd |
The brain's monthly eval budget, the run cap per call, and this calendar month's spend (UTC). Live runs, judged submissions and the Answer Key drafter all spend from it.
{ "brain_id": "calliope", "monthly_usd": 50.0, "max_runs_per_call": 10,
"spent_usd_this_month": 12.4, "remaining_usd": 37.6 }
The Advisor¶
POST /v1/brains/{brain_id}/chat/completions is in the published schema too. It is
OpenAI-compatible, needs the advisor:read scope, and uses the OpenAI error format. See
OpenAI-compatible Advisor, including how it asks follow-up questions
(needs_input) and how to reply.