Skip to content

Questions Framework API

The Questions Framework developer API lets your own system work with a brain's Questions: read and match them, propose new ones, draft Answer Keys, run and submit evals, and (with a Curator or Admin key) approve and certify. This page is the reference for every route. For concepts, flags and how to get a key, see Access, flags and scopes.

The same lifecycle is also available to agents over MCP: see Questions Framework over MCP.

OpenAPI schema

The published schema is openapi.json (OpenAPI 3.1). It covers every route on this page and the OpenAI-compatible Advisor. Load it into your API client or code generator. This page adds what the schema can't say well: who may call, which flags gate a route, and the redaction rule.

Three routes take a free-form JSON body, so the schema shows no request body for them: POST /questions/{question_id}/answers, POST /eval-results/{result_id}/grades and POST /chat/completions. Their bodies are described below and in the OpenAI-compatible Advisor page.

Base URL

Every route is under the brain it acts on:

https://<host>/v1/brains/{brain_id}/…

Locally the gateway runs on http://localhost:8088. The examples on this page use these shell variables:

export POLARIS=http://localhost:8088          # or https://<host>
export BRAIN=calliope                         # your brain id
export KEY=sk-polaris-…                       # your API key

Authentication

Send a bearer credential on every request:

Authorization: Bearer sk-polaris-…

Two kinds of credential work:

  • An API key (sk-polaris-…). A key is bound to one brain and carries a list of scopes. Each route below names the scope it needs. A key for another brain is 403 brain_forbidden. A key without the scope is 403 insufficient_scope.
  • A signed-in session (a Polaris access token). A person's role decides what they may do: reads need Reader or above, everything else needs Curator or Admin.

A key's actor is who it acts as. A key minted over REST (POST /v1/brains/{id}/api-keys) acts as the person who minted it; a key minted with bb apikey mint acts as its --actor. Proposals, drafts, approvals and sign-offs made with a key are recorded as that actor, and the audit row records the key's id.

A key's role (Reader, Curator or Admin) matters in two places. Curator acts (approve, sign off, publish a version, edit slots) need a Curator or Admin key. Rubric detail is shown only to a Curator or Admin key (see Redaction).

Curator acts by key. Besides the key's role and the act's scope, the key's actor must be an active Curator or Admin account of this brain, and not an agent identity (agent:, agent/, bot:, system:, … (agent)). This is checked every time the key acts, not only when it was minted, so:

  • a key minted by a platform operator can't do Curator acts: the operator has no Curator account in the brain;
  • a key stops doing Curator acts once its actor is deactivated, or demoted to Reader;
  • a key minted with bb apikey mint acts as --actor, so for Curator acts pass the login email of a real Curator or Admin of the brain. The CLI's default actor (apikey@polaris) has no account and is refused.

A refused actor is 403: approval_refused with detail.code actor_not_a_curator (or not_a_person) when approving, and actor_not_a_curator (or not_a_person) as the error code when signing off, starting a version or editing slots.

To get a key, and for the full list of scopes and flags, see Access, flags and scopes.

Feature flags

The API is off until a platform admin turns it on for your brain. All flags default to off.

Flag What it gates
question_store_db Every route on this page.
qf_developer_api API keys on every route except the five "DEE-147" routes (submit answers, grade, standing, list eval runs, judge agreement), which need only question_store_db. A signed-in person is not affected by this flag.
question_catalog List, match and read Questions, for anyone.
question_slot_editing Start a Question version and edit slots.
questions_workspace Certification sign-off.

A flag that is off is 403 feature_disabled, never 404, so you can tell it apart from a missing Question. The message names the flag.

Conventions

Pagination

List routes take cursor and limit query parameters and return a page:

{ "items": [ … ], "next_cursor": "eyJvIjo1MH0" }

Pass next_cursor back as cursor to get the next page. next_cursor: null means the last page. The cursor is opaque: don't build or change one. A cursor this API didn't return (including one naming an offset past 1,000,000) is 400 invalid. limit defaults to 50 (100 for GET /eval-runs) and is capped at 500.

curl -s "$POLARIS/v1/brains/$BRAIN/questions?limit=20" -H "Authorization: Bearer $KEY"
curl -s "$POLARIS/v1/brains/$BRAIN/questions?limit=20&cursor=eyJvIjoyMH0" -H "Authorization: Bearer $KEY"

Retrying writes safely (Idempotency-Key)

Every POST and PUT in the Answer Keys, Evals (runs) and Approvals groups takes an optional Idempotency-Key header. Send a unique value (a UUID is fine, up to 255 printable characters). If the request times out, send it again with the same key. You get the first response back instead of the write happening twice: one proposed Question, one drafted Answer Key, one paid eval run.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/proposals" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: 6f1c2e0a-3b7d-4c39-9a51-0d2f7b1e9c44" \
  -d '{"text": "Do we need a license to lend in {jurisdiction}?", "intent": "Decide whether to apply for a lender license before lending.", "topic": "licensing", "slots": ["jurisdiction:us_state"]}'

How it behaves:

  • A key is remembered per brain, per caller and per route, for 24 hours.
  • The response carries Idempotent-Replayed: false the first time and Idempotent-Replayed: true on a replay.
  • Only a success is remembered. A refusal (say 402 budget_exceeded) is not, so a retry asks again.
  • The same key with a different request (another path, query or body) is 409 idempotency_conflict. Use a new key for a new request.
  • A retry while the first request is still running is 409 idempotency_in_progress. Wait and retry.
  • If the server can't store keys right now, a request that sends the header is 503 idempotency_unavailable rather than running without the guarantee. A request without the header is unaffected.

POST /questions/{question_id}/answers doesn't take the header. It has its own retry rule: reuse the same external_run_id.

Errors

Every refusal uses one envelope:

{
  "error": {
    "code": "feature_disabled",
    "message": "'qf_developer_api' (with 'question_store_db') is disabled for this brain",
    "detail": null
  }
}

Branch on code. It is stable; message is for people and may change. detail carries extra fields for some codes (for example required_scope, or the refusal's own code for approval_refused).

HTTP code Meaning
400 invalid The request is wrong: a bad cursor, a missing field, a binding that doesn't fit the Question, an Idempotency-Key that is too long. message says what.
400 run_cap_exceeded More runs per call than the brain allows (see GET /budget).
401 unauthorized No Authorization: Bearer … header, or an invalid access token.
401 invalid_api_key The sk-polaris-… key doesn't exist.
401 revoked_api_key The key was revoked.
402 budget_exceeded The brain's monthly eval budget would be exceeded. Nothing was spent.
402 max_cost_exceeded The run's estimate is over the max_cost_usd you set. Nothing was spent. detail has estimate_usd and max_cost_usd.
403 insufficient_scope The key lacks the route's scope. detail.required_scope names it.
403 brain_forbidden The key or session is for another brain.
403 role_forbidden A Curator act with a Reader key, or a role that may not use the route.
403 feature_disabled A Questions Framework flag is off for this brain. message names it.
403 forbidden Not allowed for this caller: a platform operator approving a tenant's work, a key grading as someone else, an agent identity grading.
403 actor_not_a_curator, not_a_person A Curator act (sign off, start a version, edit slots, save a rubric, approve a slot type) by a key whose actor isn't an active Curator or Admin account of this brain, or is an agent identity. See Authentication.
403 approval_refused An approval or withdrawal refused because of who is asking. detail.code is not_a_person, not_the_approver, self_approval, approver_required, not_the_requester or actor_not_a_curator.
404 not_found No such Question, Answer Key, run, result, certification or eval op.
409 conflict An id you chose is taken: a question_id on a proposal, an answer_key_id on a draft, or a slot type's name.
409 approval_refused Refused because of the request's state. detail.code says why, for example no_draft, already_requested, approver_unknown, not_pending, expired, draft_changed, gate_failed.
409 citations_missing A rubric criterion cites no provision, or one the corpus doesn't hold. detail.citations lists each check.
409 not_eligible, invalidated, already_signed, not_the_published_version A sign-off refused: the certification is no longer eligible, was invalidated, is already signed, or is for an older brain version.
409 seed_unpinned, no_brain_version A live run refused before spending: the server has no fixed PYTHONHASHSEED, or the brain has no corpus at the ref. detail.remedy may say what to do.
409 idempotency_conflict, idempotency_in_progress See Retrying writes safely.
422 invalid_request The body, path or query failed validation (see below).
503 judge_unavailable The judge model is not available. Retry later, or score without the judge.
503 idempotency_unavailable An Idempotency-Key was sent and can't be honoured right now.

Validation errors

A body, path or query parameter that fails validation is 422 invalid_request in the same envelope. detail.errors lists each problem:

{
  "error": {
    "code": "invalid_request",
    "message": "the request failed validation",
    "detail": {
      "errors": [
        { "loc": ["body", "slot"], "msg": "Extra inputs are not permitted", "type": "extra_forbidden" }
      ]
    }
  }
}

Request bodies refuse fields they don't know, so a misspelt field is a 422, not silently ignored. Responses may gain fields over time: ignore fields you don't know.

What is never returned (redaction)

  • Answer text is never returned by any route. Eval results carry scores, never the answer.
  • Rubric detail is shown only to a caller who may see rubrics: a Curator or Admin person, a platform admin, or an API key that has questions:read and a Curator or Admin role. Rubric detail is an Answer Key's rubric, the drafter's citations, and a result's per-criterion fields (parts, judged_parts, penalties_fired, ungraded, capped_by, human_grades).
  • Everyone else gets rubric: null, no per-criterion fields, and a boolean capped in place of the penalty wording. GET /questions/{question_id} says which you got in rubric_visible.
  • Judge agreement names criteria by key only, never their wording.

Questions

Read the brain's published Questions, match an ask to one, and read slots, standing, certification and coverage. All need questions:read (a key) or Reader and above (a person).

List published Questions

GET /v1/brains/{brain_id}/questions

Scope questions:read
Flags question_catalog; a key also needs question_store_db and qf_developer_api
Query cursor, limit, topic (one topic card id)
Returns A page of QuestionSummary: question_id, version, status, text, intent, topic, slots
curl -s "$POLARIS/v1/brains/$BRAIN/questions?topic=licensing" -H "Authorization: Bearer $KEY"
{
  "items": [
    {
      "question_id": "Q-lender-license",
      "version": 1,
      "status": "published",
      "text": "Do we need a license to lend in {jurisdiction}?",
      "intent": "Decide whether to apply for a lender license before lending.",
      "topic": "licensing",
      "slots": [
        { "name": "jurisdiction", "type": "us_state", "required": true,
          "ask_text": "Which state will you lend in?" }
      ]
    }
  ],
  "next_cursor": null
}

A slot's {name} appears in the text only when the slot is required.

Match an ask to a Question

POST /v1/brains/{brain_id}/questions/match

Which published Question a person's question is. No model call.

Scope questions:read
Flags as for list
Body question (required, the ask as typed), topic_ids (optional, topic card ids to narrow to first)
Returns status (matched, uncertain or none), question_id, candidates (question_id, score, topic), binding, binding_problems, scoped_to
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/match" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"question": "Do I need a lender license to make loans in Texas?"}'
{
  "status": "matched",
  "question_id": "Q-lender-license",
  "candidates": [{ "question_id": "Q-lender-license", "score": 0.91, "topic": "licensing" }],
  "binding": { "jurisdiction": "US-TX" },
  "binding_problems": [],
  "scoped_to": []
}

binding holds the slot values the ask states. Check binding_problems before you use it. On uncertain, ask the person which candidate they meant; never bind on uncertain. An empty question is 422.

Read a Question

GET /v1/brains/{brain_id}/questions/{question_id}

Scope questions:read
Flags as for list
Returns The summary fields, plus required_provisions (always, by_binding), answer_shape, certified_bindings (binding keys), answer_keys, rubric_visible

Each Answer Key has answer_key_id, version, is_draft, status, binding, binding_key, content_sha, drafted_by, approved_by, approved_at and rubric (null unless you may see rubrics). Errors: 404 not_found.

curl -s "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license" -H "Authorization: Bearer $KEY"

A binding key is the canonical form of a binding: slot=value pairs sorted by slot and joined by &, for example jurisdiction=US-TX&product=loan. A Question with no slots has the empty binding key "".

A Question's slots

GET /v1/brains/{brain_id}/questions/{question_id}/slots

Scope questions:read
Flags question_store_db; a key also needs qf_developer_api
Returns question_id, published and draft (each version, text, slots, status, change_reason; draft is null with no open draft), bindings (answer_key_id, binding, binding_key, state, question_version)

Read this before PUT /questions/{question_id}/slots: you pass back the version you read. Errors: 404 not_found.

Slot types

GET /v1/brains/{brain_id}/slot-types

Scope questions:read
Flags as for slots
Query include_proposed (default false): also the brain's proposed types
Returns A list of name, kind (parameter or applicability), base (enum, string, corpus_ref, number, date), namespace, description, values (an enum's values), provision_key_prefix, status (active or proposed), origin (platform or the brain id), proposed_by, approved_by

The slot types a Question's slots may use: the platform's and the brain's own. Use them to build a binding. A proposed type is listed only with include_proposed, and no slot may use it until it is approved (Propose a slot type).

A Question's standing over time

GET /v1/brains/{brain_id}/questions/{question_id}/standing

Scope questions:read
Flags question_store_db only
Query answer_key_id, binding_key, and tuple filters model, brain_version, ref, seed, judge_model, source; end_customer_id
Returns brain_id, question_id, points: [{at, standing, run_ids, tuple}], oldest first

question_id may also be a standalone Answer Key's id. Each point carries its comparability tuple (Question and Answer Key version, brain version, model, judge model, seed, ref, source). Compare only points whose tuples are equal: a change in any field measures something else. end_customer_id keeps only points whose runs were all submitted for that end customer.

curl -s "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/standing?source=live&ref=published" \
  -H "Authorization: Bearer $KEY"

Certified bindings

GET /v1/brains/{brain_id}/questions/{question_id}/certified-bindings

Scope questions:read
Flags question_store_db; a key also needs qf_developer_api
Returns brain_id, question_id, items: each binding_key, answer_key_id, answer_key_version, brain_version, standing, threshold, evidence_run_ids, computed_at, signed_by, signed_at

The bindings certified at the brain's published version, with their evidence runs. A brain that never published has nothing certified. Errors: 404 not_found.

Question coverage

GET /v1/brains/{brain_id}/question-coverage

GET /v1/brains/{brain_id}/question-coverage/check?question_id=…&binding=…

GET /v1/brains/{brain_id}/question-coverage/check?answer_key_id=…

Scope questions:read
Flags a key needs question_store_db and qf_developer_api; a person needs none
Returns The matrix: brain_id, corpus_empty, summary, rows. A check: one row plus brain_id and corpus_empty

Whether the published corpus holds the provisions each Question needs at each binding. Each row has question_id, binding, answer_keys, status (covered, gap or undeclared), held, required, missing, provisions and unroutable_criteria. No model call.

binding is the binding key, URL-encoded:

curl -s -G "$POLARIS/v1/brains/$BRAIN/question-coverage/check" \
  --data-urlencode "question_id=Q-lender-license" \
  --data-urlencode "binding=jurisdiction=US-TX" \
  -H "Authorization: Bearer $KEY"

Errors: 400 invalid (a binding part that isn't slot=value), 404 not_found.

Answer Keys

Propose a Question and draft its Answer Key. With a Curator or Admin key, approve it, start a new Question version, and edit slots. Every route here takes an Idempotency-Key.

Propose a Question

POST /v1/brains/{brain_id}/questions/proposals → 201

Scope questions:write
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Body text (required, up to 4000 characters; a slotted Question names each slot as {slot}), intent (required, one sentence on what the asker needs to decide), topic (required, a topic card id), slots (each name, name:type, or an object {name, type, required, ask_text, default}), question_id (optional, your own id: Q- then lower-case letters, digits and -), required_provisions (optional, {always: [key…], by_binding: {binding_key: [key…]}}), answer_shape (optional, as GET /questions/{question_id} returns it)
Returns question_id, version, is_draft, store_commit, next_step
curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/proposals" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"text": "Do we need a license to lend in {jurisdiction}?",
       "intent": "Decide whether to apply for a lender license before lending.",
       "topic": "licensing", "slots": ["jurisdiction:us_state"]}'
{
  "question_id": "Q-do-we-need-a-license-to-lend-3fa9c2",
  "version": 1,
  "is_draft": true,
  "store_commit": "9c1e…",
  "next_step": "draft its Answer Key: POST /questions/{question_id}/answer-keys/drafts; a Curator publishes the Question"
}

Polaris generates the question_id unless you give one. Giving it keeps ids the same when you copy Questions from one brain to another; it is 409 conflict when the brain already has that id. The Question is a draft. It is not served, scored or certified until a Curator publishes it. No model call. Errors: 400 invalid (for example a slot named in the text but not declared, or a slot type that is only proposed), 409 conflict.

Draft an Answer Key

POST /v1/brains/{brain_id}/questions/{question_id}/answer-keys/drafts → 201

Scope questions:write
Role (person) Curator or Admin
Flags as for propose
Body binding (a value for each required slot), answer (an expert's reference answer), use_drafter (default false), answer_key_id (optional, your own id)
Returns answer_key_id, version, agent_drafted, store_commit, estimate_usd, rubric, citations

Drafts the Answer Key for one binding of the Question. You have three options:

  • answer with use_drafter: true: the drafter proposes the rubric from your answer (one model call).
  • use_drafter: true without answer: the drafter writes the reference answer from the corpus first (two model calls).
  • answer with use_drafter: false: a one-criterion skeleton of your answer. Free.

Drafter calls are checked against the brain's monthly budget before the first one, and recorded against it. estimate_usd is what they were priced at (null when no model was called). rubric and citations are null unless you may see rubrics.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/answer-keys/drafts" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"binding": {"jurisdiction": "US-TX"},
       "answer": "Yes. Texas requires a license under Tex. Fin. Code § 342.051 for…",
       "use_drafter": true}'

The draft scores nothing until it is approved. To give it a whole rubric rather than the skeleton, save one over it with a Curator key (Save a whole rubric). Next, ask a Curator to approve it (POST /approvals) or approve it yourself with a Curator key (POST /answer-keys/{id}/approve).

answer_key_id names the Answer Key instead of having Polaris mint one (409 conflict when it is taken). A draft Question's first Answer Key is the Question's own draft, named as the Question's id without Q-, so for that one answer_key_id must be that name.

Errors: 400 invalid (no answer and use_drafter: false; a binding that doesn't fit the Question's slots; a binding that already has an Answer Key), 402 budget_exceeded, 404 not_found, 409 conflict.

Save a whole rubric (Curator key)

PUT /v1/brains/{brain_id}/answer-keys/{answer_key_id}/draft

Scope questions:write, on a Curator or Admin key
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Body rubric (required, as GET /questions/{question_id} returns it to a Curator in answer_keys[].rubric), change_reason (needed over an approved Answer Key: <kind> or <kind>: <note>, kind one of law_changed, new_adjudication, scope_refined, error_corrected)
Returns answer_key_id, version, content_sha, validation (ok, errors, citations), store_commit

Saves a rubric with all its parts as the Answer Key's draft: criteria with their weights, patterns and cited provisions, verdicts, gold, gold_when and penalties. An open draft is replaced at its version. Over an approved Answer Key, the save is the next version, which needs change_reason; the approved version keeps scoring until the new one is approved. The Answer Key's id, version, binding and Question are the server's; whatever the rubric says about them is ignored.

A rubric that doesn't load is 400 invalid. One that loads but fails a check is saved, and validation says why approval would refuse it yet: a failing gate in errors, a criterion with a missing or unresolved provision in citations. No model call.

Only a Curator or Admin key may save a rubric, because only they may read one, and the key's actor must be an active Curator or Admin account of this brain (403 actor_not_a_curator).

curl -s -X PUT "$POLARIS/v1/brains/$BRAIN/answer-keys/lender-license-tx/draft" \
  -H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" -d @rubric.json

Errors: 400 invalid, 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person, 404 not_found.

Propose a slot type

POST /v1/brains/{brain_id}/slot-types → 201

Scope questions:write
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Body name (required, lower-case letters, digits and _), base (required: enum, string, corpus_ref, number, date), namespace (required, how org data is segmented, for example product or timing), description (required), values (an enum's values), kind (parameter, the default, or applicability)
Returns The slot type, status: proposed

Adds one of the brain's own slot types, for slots no platform type fits. It is proposed: no slot may use it until someone other than its proposer approves it. 409 conflict when the name is taken.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/slot-types" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"name": "credit_purpose", "base": "enum", "namespace": "product",
       "description": "Consumer-purpose or business-purpose credit.",
       "values": ["consumer", "business"]}'

Approve a slot type (Curator key)

POST /v1/brains/{brain_id}/slot-types/{name}/approve

Scope slots:write, on a Curator or Admin key
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Body note (optional, recorded in the store's commit)
Returns The slot type, status: active

Makes a proposed slot type usable. The key's actor is the approver, must be an active Curator or Admin account of this brain, and must not be the proposer (403 approval_refused, detail.code: self_approval). Errors: 403 role_forbidden, 403 actor_not_a_curator, 404 not_found (no such proposed type).

Approve an Answer Key (Curator key)

POST /v1/brains/{brain_id}/answer-keys/{answer_key_id}/approve

Scope answer_key:approve, on a Curator or Admin key
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Body note (optional, recorded in the audit log)
Returns answer_key_id, approved_by, version, commit, api_key_id

Approves the Answer Key's current draft directly, with no request needed. It runs the same checks as an approval in the web app: every criterion must cite a provision that resolves, and the draft must pass validation. The key's actor is the approver, and the audit row records the key.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/answer-keys/lender-license-tx/approve" \
  -H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" -d '{"note": "Checked against § 342.051."}'

An approval is a curator's claim about what counts as a right answer. Approve only a rubric a person has read.

Errors: 403 role_forbidden (a Reader key, whatever its scopes), 403 approval_refused with detail.code: not_a_person (the key's actor is an agent identity) or actor_not_a_curator (the key's actor isn't an active Curator or Admin account of this brain, for example a key a platform operator minted), 403 forbidden (a platform operator's session), 409 approval_refused (for example no_draft, already_approved, gate_failed), 409 citations_missing, 404 not_found.

Start a new Question version (Curator key)

POST /v1/brains/{brain_id}/questions/{question_id}/versions → 201

Scope questions:publish, on a Curator or Admin key
Role (person) Curator or Admin
Flags question_store_db and question_slot_editing; a key also needs qf_developer_api
Body text, slots (each name, type, required, ask_text, default), change_reason (required), fill (a value for each added required slot), drop (removed slots whose bound values may go), dry_run
Returns question_id, version, applied, changes, problems, warnings, dropping, store_commit, answer_key_drafts, skipped

Starts version N+1 of a published Question with new text or slots. Each binding is carried to the new version as a draft Answer Key (answer_key_drafts; skipped says which weren't and why). Version N stays live until the first of those drafts is approved. dry_run: true reports the plan without writing.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/versions" \
  -H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"text": "Do we need a license to make {credit_purpose} loans in {jurisdiction}?",
       "slots": [{"name": "jurisdiction", "type": "us_state"},
                 {"name": "credit_purpose", "type": "credit_purpose"}],
       "change_reason": "Licensing differs for consumer and business credit.",
       "fill": {"credit_purpose": "consumer"}, "dry_run": true}'

Errors: 400 invalid, 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person (the key's actor), 403 feature_disabled, 404 not_found.

Edit slots (Curator key)

PUT /v1/brains/{brain_id}/questions/{question_id}/slots

Scope slots:write, on a Curator or Admin key
Role (person) Curator or Admin
Flags as for versions
Body version (the version you read with GET …/slots), text, slots, fill, drop, dry_run
Returns question_id, version, applied, changes, problems, warnings, dropping, store_commit

Edits the slots of a draft Question version, or only the ask text (ask_text) of a published one. Anything else on a published Question needs a new version. dry_run: true reports without writing. Errors: 400 invalid, 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person (the key's actor), 403 feature_disabled, 404 not_found.

Evals

Estimate and run live evals within the brain's budget, submit answers your own system produced, record grades, and read runs.

Estimate a run

GET /v1/brains/{brain_id}/questions/{question_id}/runs/estimate

Scope eval:run
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Query runs (answers per Answer Key, default 1), judge (default true), answer_key_id (one binding; else every binding)
Returns estimate_usd, remaining_usd, monthly_usd, allowed, reason

What POST …/runs would cost, priced exactly as the run is authorized. Free. It never refuses over budget: allowed: false and reason say the run would be refused.

curl -s "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/runs/estimate?runs=5" \
  -H "Authorization: Bearer $KEY"

Start a live eval run

POST /v1/brains/{brain_id}/questions/{question_id}/runs → 202

Scope eval:run
Role (person) Curator or Admin
Flags as for estimate
Body runs (default 1), judge (default true), ref (draft or published, default draft), answer_key_id (one binding), max_cost_usd (refuse if the estimate is higher)
Returns op_id, status (running or done), estimate_usd, run_ids

This costs money. The Advisor answers each binding runs times, and Polaris judges, scores and stores every answer. The budget, the run cap and your max_cost_usd are checked before any model call. ref: draft scores the corpus as it stands; only ref: published runs can certify the published version. Without judge, the judged criteria are ungraded and the run can't certify.

A run takes minutes, so the request returns 202 as soon as the run is past every refusal and spending. Then poll the op.

The eval run flow

  1. Start the run and keep the op_id:

    curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/runs" \
      -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
      -H "Idempotency-Key: $(uuidgen)" \
      -d '{"runs": 5, "ref": "published", "max_cost_usd": 2.00}'
    
    { "op_id": "op_5d0c…", "status": "running", "estimate_usd": 0.5, "run_ids": [] }
    
  2. Poll GET /v1/brains/{brain_id}/eval-ops/{op_id} until status is done or error:

    curl -s "$POLARIS/v1/brains/$BRAIN/eval-ops/op_5d0c…" -H "Authorization: Bearer $KEY"
    
    {
      "op_id": "op_5d0c…",
      "status": "done",
      "progress": 1.0,
      "detail": "…",
      "run_ids": [11, 12],
      "error": null,
      "result": { "…": "the run's report: standing per binding, spend, certification changes" }
    }
    
  3. Read each run with GET /v1/brains/{brain_id}/eval-runs/{run_id}.

Notes on polling:

  • GET /eval-ops/{op_id} needs eval:run. Only whoever started the run can see it; anyone else gets 404. For a key that means the same key: another key is 404 even if it has the same actor. An op is held in the gateway's memory, so after a gateway restart it is 404 too: find the run in GET /eval-runs instead.
  • If an Advisor or judge call fails part-way, the answers already produced are still stored. result.code is then partial (or persist_failed) and error says what went wrong.
  • A retry with the same Idempotency-Key returns the same op_id and starts nothing. If the first request was cut off before it answered, a retry is 409 idempotency_in_progress until that key's pending record expires (15 minutes), never a second run: find the run in GET /eval-runs.

Errors on start: 400 run_cap_exceeded, 402 budget_exceeded, 402 max_cost_exceeded, 404 not_found, 409 seed_unpinned, 409 no_brain_version, 503 judge_unavailable.

Submit answers to score

POST /v1/brains/{brain_id}/questions/{question_id}/answers → 201 (or 200 for a duplicate)

Scope eval:submit; also eval:run when judge is true
Role (person) Curator or Admin
Flags question_store_db only
Returns duplicate, run_ids, runs, standing, comparability, authorization

Send answers your own system produced. Polaris scores them against the Question's rubric and stores them as source: external. question_id may also be a standalone Answer Key's id.

The body is a JSON object:

Field Required Meaning
answers yes Up to 200 of {"answer_key_id": "…", "text": "…"}.
tuple yes How the answers were produced: {"model", "brain_version", "ref", "seed"}. All four are needed: a score without them can't be compared with anything. ref is draft or published.
judge no true also grades the judged criteria. That spends the eval budget and needs eval:run.
external_run_id no Your id for this submission. Resubmitting it returns the stored runs with duplicate: true (HTTP 200) and writes nothing.
end_customer_id no Which of your customers the answers were for (up to 128 characters). Filter on it later.

Never send a score: a field named score, points, share, verdict, judged or passed on the submission or an answer is 400 invalid.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/questions/Q-lender-license/answers" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"answers": [{"answer_key_id": "lender-license-tx", "text": "Yes. Under Tex. Fin. Code § 342.051…"}],
       "tuple": {"model": "acme:gpt-x", "brain_version": "147676ad", "ref": "published", "seed": "0"},
       "external_run_id": "acme-batch-2026-11-07-001"}'

External runs show in standing but never certify. Errors: 400 invalid, 402 budget_exceeded, 404 not_found, 503 judge_unavailable.

Grade a result

POST /v1/brains/{brain_id}/eval-results/{result_id}/grades

Scope grades:submit
Who A named person. A signed-in Curator or Admin grades as themself. A key grades only as its own actor, and only if the key's role is Curator or Admin and its actor is an active Curator or Admin of this brain (not an agent identity).
Flags question_store_db only
Body criterion (a judged criterion's key), satisfied (true or false), note (optional), grader (optional; if given, it must be the key's own actor)
Returns The result, rescored, with tuple and run_standing

A person's grade overrides the judge for that criterion on that result, and the run is rescored. The result_id is a result's id from GET /eval-runs/{run_id}. Only grade a criterion a person has read the answer for.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/eval-results/4021/grades" \
  -H "Authorization: Bearer $CURATOR_KEY" -H "Content-Type: application/json" \
  -d '{"criterion": "C3_right", "satisfied": true, "note": "Cites the right section."}'

Errors: 400 invalid (satisfied not a boolean, an unknown criterion), 403 forbidden (a key naming another grader, an agent identity, a Reader key), 404 not_found.

List eval runs

GET /v1/brains/{brain_id}/eval-runs

Scope questions:read
Flags question_store_db only
Query question_id, answer_key_id, source (live, external, file, reference), end_customer_id, cursor, limit (default 100)
Returns A page of runs, newest first

Each run has id, its comparability tuple fields (question_id, question_version, answer_key_id, answer_key_version, model, judge_model, brain_version, ref, seed, source), binding_key, standing, rubric_approved, provenance (submitted_by, api_key_id, external_run_id, end_customer_id), content_sha, cost_usd and created_at.

Read an eval run

GET /v1/brains/{brain_id}/eval-runs/{run_id}

Scope questions:read
Flags question_store_db; a key also needs qf_developer_api
Returns The run, plus results: each id, run_id, run_index, ok, error, verdict, share, full_credit, and capped or the rubric detail

Note the flags differ from the list: GET /eval-runs is a DEE-147 route and needs only question_store_db, while GET /eval-runs/{run_id} is part of the developer API, so a key also needs qf_developer_api (a signed-in person doesn't).

{
  "id": 11,
  "question_id": "Q-lender-license",
  "answer_key_id": "lender-license-tx",
  "binding_key": "jurisdiction=US-TX",
  "source": "live",
  "ref": "published",
  "standing": 0.86,
  "cost_usd": 0.1,
  "results": [
    { "id": 4021, "run_id": 11, "run_index": 0, "ok": true, "verdict": "YES",
      "share": 0.86, "full_credit": false, "capped": false }
  ]
}

Errors: 404 not_found.

Judge agreement

GET /v1/brains/{brain_id}/judge-agreement

Scope questions:read
Flags question_store_db only
Query judge_model, question_id
Returns brain_id, criteria: each answer_key_id, criterion, judge_model, both, agreed, rate

How often the judge's call matched a person's grade, per judged criterion. Criteria are named by key, never by their wording.

Approvals

Ask a named Curator to approve a draft, withdraw a request, and sign off certifications. Every POST here takes an Idempotency-Key.

List pending approval requests

GET /v1/brains/{brain_id}/approvals

Scope questions:write
Role (person) Curator or Admin
Flags question_store_db; a key also needs qf_developer_api
Query status (only pending), cursor, limit
Returns A page of Approval: id, answer_key_id, status, requested_by, approver, note, created_at, expires_at, decided_by, decided_at

With a key: the pending requests its actor asked for or is named on, oldest first, without the draft. A signed-in person sees the requests that name them (an Admin sees every one), with the draft attached for the named approver. Errors: 400 invalid (another status).

Request an approval

POST /v1/brains/{brain_id}/approvals

Scope questions:write
Role (person) Curator or Admin
Flags as for list (a person also needs questions_workspace)
Body answer_key_id, approver (the login email of a Curator or Admin of this brain)
Returns The pending Approval

Asks a named person to approve a draft Answer Key. They decide in the web app, or approve directly with their own Curator key. The request is recorded as asked by the key's actor. A request expires after 14 days, and a draft edited after the request can't be approved under it.

curl -s -X POST "$POLARIS/v1/brains/$BRAIN/approvals" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"answer_key_id": "lender-license-tx", "approver": "curator@acme.example"}'

Errors: 409 citations_missing (a criterion is uncited or cites a provision that doesn't resolve, where the Questions workspace is on), 409 approval_refused (no_draft, already_requested, approver_unknown), 422 invalid_request.

Withdraw an approval request

POST /v1/brains/{brain_id}/approvals/{pending_id}/withdraw

Scope questions:write
Role (person) Curator or Admin (the requester, or an Admin)
Flags as for list (a person also needs questions_workspace)
Body note (optional)
Returns The withdrawn Approval (status: superseded)

A key may withdraw only a request its own actor made (403 approval_refused, detail.code: not_the_requester). A decided request is 409 approval_refused (not_pending).

Deciding a request (approve or reject) is not on the developer API: the named person decides in the web app. A Curator or Admin key approves directly with POST /answer-keys/{id}/approve.

Sign off a certification (Curator key)

POST /v1/brains/{brain_id}/certifications/{certification_id}/sign-off

Scope certification:sign_off, on a Curator or Admin key
Role (person) Curator or Admin
Flags question_store_db and questions_workspace; a key also needs qf_developer_api
Body note (optional)
Returns certification_id, signed_by, signed_at, api_key_id

Signs off an eligible certification at the published version, so the binding is certified. Eligibility is judged again first. The key's actor is the signer, and the audit row records the key. Where questions_workspace is off, a certification is trusted without a sign-off, so this route is 403 feature_disabled.

Errors: 403 role_forbidden, 403 actor_not_a_curator / 403 not_a_person (the key's actor), 403 forbidden (a platform operator's session), 404 not_found, 409 not_eligible, 409 invalidated, 409 already_signed, 409 not_the_published_version.

Budget

The eval budget

GET /v1/brains/{brain_id}/budget

Scope questions:read
Flags question_store_db; a key also needs qf_developer_api
Returns brain_id, monthly_usd, max_runs_per_call, spent_usd_this_month, remaining_usd

The brain's monthly eval budget, the run cap per call, and this calendar month's spend (UTC). Live runs, judged submissions and the Answer Key drafter all spend from it.

curl -s "$POLARIS/v1/brains/$BRAIN/budget" -H "Authorization: Bearer $KEY"
{ "brain_id": "calliope", "monthly_usd": 50.0, "max_runs_per_call": 10,
  "spent_usd_this_month": 12.4, "remaining_usd": 37.6 }

The Advisor

POST /v1/brains/{brain_id}/chat/completions is in the published schema too. It is OpenAI-compatible, needs the advisor:read scope, and uses the OpenAI error format. See OpenAI-compatible Advisor, including how it asks follow-up questions (needs_input) and how to reply.