Walkthrough: the Questions Framework with an API key¶
This page takes you through the Questions Framework (QF) end to end, as an outside developer holding an API key. You list a brain's Questions, match an ask to one, propose a new Question, draft its Answer Key, get it approved, run a live eval, read the results and the standing, ask the Advisor a question that it routes to a Question, and do the same reads over MCP.
Every step is also a function in examples/qf_quickstart.py, in
the same order. Run it to see the whole flow against your gateway (see
Run the quickstart). A CI test runs the same functions against an in-process
gateway, so the shapes below are the ones the code returns.
For the full reference, see:
- Questions API reference: every route, field and error.
- MCP: the remote QF MCP endpoint and its tools.
- Access: flags, scopes and roles in detail.
- Questions Framework overview: what Questions, Answer Keys, runs and standing are.
Prerequisites¶
Flags a platform admin turns on¶
QF features are behind per-brain feature flags. Only a platform admin can change them, in the web
app (Tenant settings > Feature flags) or with PUT /v1/brains/{brain_id}/flags/{flag_key}.
| Flag | Needed for |
|---|---|
question_store_db |
Every QF route. |
qf_developer_api |
Every QF route when the caller is an API key. |
question_catalog |
GET /questions, POST /questions/match, GET /questions/{id}. |
questions_workspace |
Certification sign-off and status. With it on, approval also requires every criterion to cite a provision that resolves. |
question_slot_editing |
POST /questions/{id}/versions, PUT /questions/{id}/slots. |
advisor_question_routing |
The Advisor matching an ask to a Question and asking for missing facts. |
A platform admin sets a flag like this (with a platform admin's JWT):
curl -X PUT "$POLARIS_URL/v1/brains/$POLARIS_BRAIN/flags/qf_developer_api" \
-H "Authorization: Bearer $ADMIN_JWT" -H "Content-Type: application/json" \
-d '{"value": true}'
The Advisor also needs the brain's own OpenRouter key. A tenant Admin adds it in the web app under
Settings > Model providers. Without it, /chat/completions returns 403 openrouter_key_required.
Mint the keys¶
A tenant Admin (or a platform admin) mints keys with their JWT. Get a JWT with
POST /v1/auth/login:
ADMIN_JWT=$(curl -s -X POST "$POLARIS_URL/v1/auth/login" -H "Content-Type: application/json" \
-d '{"email": "admin@example.com", "password": "..."}' | jq -r .access_token)
Mint a Reader key for your integration. It reads, proposes, drafts, runs evals and asks the Advisor:
curl -X POST "$POLARIS_URL/v1/brains/$POLARIS_BRAIN/api-keys" \
-H "Authorization: Bearer $ADMIN_JWT" -H "Content-Type: application/json" \
-d '{"label": "acme-integration", "role": "Reader",
"scopes": ["questions:read", "questions:write", "eval:run", "advisor:read"]}'
{"id": 12, "brain_id": "meridian", "role": "Reader", "actor": "admin@example.com",
"scopes": ["questions:read", "questions:write", "eval:run", "advisor:read"],
"revoked": false, "key": "sk-polaris-..."}
The raw key is shown once. Keep it secret. Scopes are fixed when the key is minted; to change them,
mint a new key and revoke the old one (DELETE /v1/brains/{brain_id}/api-keys/{key_id}).
If you also want to approve Answer Keys from code, mint a Curator key with
answer_key:approve. Curator acts (answer_key:approve, certification:sign_off,
questions:publish, slots:write) need a key with role Curator or Admin; minting refuses them on
a Reader key. The key's actor (whoever minted it) is recorded as the approver, so a Curator should
mint their own:
curl -X POST "$POLARIS_URL/v1/brains/$POLARIS_BRAIN/api-keys" \
-H "Authorization: Bearer $CURATOR_OR_ADMIN_JWT" -H "Content-Type: application/json" \
-d '{"label": "approvals", "role": "Curator", "scopes": ["questions:read", "answer_key:approve"]}'
A key is bound to its brain. Using it on another brain's path is 403 brain_forbidden.
Set up your shell¶
export POLARIS_URL=http://localhost:8088
export POLARIS_BRAIN=meridian
export POLARIS_KEY=sk-polaris-... # the Reader key
export POLARIS_CURATOR_KEY=sk-polaris-... # optional: the Curator key
B="$POLARIS_URL/v1/brains/$POLARIS_BRAIN"
AUTH="Authorization: Bearer $POLARIS_KEY"
Every refusal comes back as:
Branch on code; it is stable. See Troubleshooting.
1. List the published Questions¶
{"items": [
{"question_id": "Q-rate-cap", "version": 1, "status": "published",
"text": "What is the maximum rate on a consumer loan in {jurisdiction}?",
"intent": "", "topic": "rates",
"slots": [{"name": "jurisdiction", "type": "us_state", "required": true,
"ask_text": null, "default": null}]}],
"next_cursor": null}
Lists are paged. Pass next_cursor back as cursor until it is null. Narrow to one topic with
?topic=rates. Read one Question, with its Answer Keys, with GET /questions/{question_id}. A
Reader key gets each Answer Key's rubric as null; only a Curator or Admin key with
questions:read sees rubrics.
Next: match what a person asked to one of these.
2. Match an ask to a Question¶
curl -s -X POST "$B/questions/match" -H "$AUTH" -H "Content-Type: application/json" \
-d '{"question": "What is the maximum rate on a consumer loan?"}'
{"status": "matched", "question_id": "Q-rate-cap",
"candidates": [{"question_id": "Q-rate-cap", "score": 1.0, "topic": "rates"}],
"binding": {},
"binding_problems": ["binding leaves slots unbound: ['jurisdiction']"],
"scoped_to": []}
status is matched, uncertain or none. On uncertain, ask the person which candidate they
meant; never bind. binding holds the slot values the ask states; check binding_problems before
you use it. Matching is free: no model call.
Next: if no Question fits, propose one.
3. Propose a draft Question¶
Writes take an optional Idempotency-Key header. Send a fresh one per request; a retry with the
same key returns the first response (with Idempotent-Replayed: true) and acts once.
curl -s -X POST "$B/questions/proposals" -H "$AUTH" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"text": "Is a late fee capped on a consumer loan?",
"intent": "Whether state law limits the late fee a lender may charge",
"topic": "fees", "slots": []}'
201:
{"question_id": "Q-is-a-late-fee-capped-on-870933", "version": 1, "is_draft": true,
"store_commit": "3fc339a0...",
"next_step": "draft its Answer Key: POST /questions/{question_id}/answer-keys/drafts; a Curator publishes the Question"}
The Question is a draft. Nobody is served or scored on it yet. A slotted Question names each slot in
its text as {slot} and declares it in slots as name or name:type. No model call.
Next: draft the Answer Key that says what a right answer is.
4. Draft its Answer Key¶
curl -s -X POST "$B/questions/Q-is-a-late-fee-capped-on-870933/answer-keys/drafts" \
-H "$AUTH" -H "Content-Type: application/json" -H "Idempotency-Key: $(uuidgen)" \
-d '{"binding": {}, "answer": "PERMITTED within limits: the cap depends on the state statute.",
"use_drafter": false}'
201:
{"answer_key_id": "is-a-late-fee-capped-on-870933", "version": 1, "agent_drafted": false,
"store_commit": "bad4931c...", "estimate_usd": null, "rubric": null, "citations": null}
use_drafter: false(the default) writes a one-criterion skeleton of your expertanswer. Free.use_drafter: truehas the drafter propose the rubric fromanswer: one model call, or two if you leaveanswerout and it writes the reference answer from the corpus. The calls are checked against the brain's monthly budget first (402 budget_exceeded) andestimate_usdsays what they were priced at.bindinggives a value for each required slot. One Answer Key per binding.rubricandcitationsarenullunless your key may see rubrics (Curator or Admin withquestions:read).
Next: a Curator approves the draft.
5. Get it approved¶
An Answer Key scores nothing until a Curator or Admin approves it. You have two ways.
Request approval from a named Curator (their login email). They decide in the web app:
curl -s -X POST "$B/approvals" -H "$AUTH" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"answer_key_id": "is-a-late-fee-capped-on-870933", "approver": "curator@example.com"}'
{"id": 1, "answer_key_id": "is-a-late-fee-capped-on-870933", "status": "pending",
"requested_by": "admin@example.com", "approver": "curator@example.com",
"created_at": "2026-11-07T15:31:48+00:00", "expires_at": "2026-11-21T15:31:48+00:00",
"decided_by": null, "decided_at": null}
requested_by is the key's actor. List your requests with GET /approvals; withdraw one with
POST /approvals/{id}/withdraw.
Or approve directly with a Curator key (answer_key:approve):
curl -s -X POST "$B/answer-keys/is-a-late-fee-capped-on-870933/approve" \
-H "Authorization: Bearer $POLARIS_CURATOR_KEY" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" -d '{"note": "checked against the statute"}'
{"answer_key_id": "is-a-late-fee-capped-on-870933", "approved_by": "curator@example.com",
"version": 1, "commit": "bb27f5a7...", "api_key_id": 13}
The key's actor is the approver, and the audit log records the key's id and your note. Approving
runs the same checks as the web app; a draft that fails them is 409. An approval refused because
of who is approving (an agent identity, a platform operator) is 403.
Approving the first Answer Key of a draft Question publishes that Question: it shows up in
GET /questions from then on.
Next: measure how well the Advisor answers a Question.
6. Estimate and start a live eval¶
A live eval has the Advisor answer each binding of a Question runs times; Polaris judges, scores
and stores every run. It costs money, so price it first. The estimate is free:
{"estimate_usd": 0.36936, "remaining_usd": 49.63, "monthly_usd": 50.0, "allowed": true, "reason": null}
Start it with a ceiling, so it is refused if the estimate is higher (402 max_cost_exceeded):
curl -s -X POST "$B/questions/Q-rate-cap/runs" -H "$AUTH" -H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" -d '{"runs": 1, "judge": true, "max_cost_usd": 0.50}'
202:
ref is draft (the corpus as it stands) by default; only published runs can certify the
published version. answer_key_id limits the run to one binding. The run is checked against the
monthly budget (402 budget_exceeded) and the run cap (400 run_cap_exceeded) before any model call.
7. Poll the run¶
While it runs:
{"op_id": "op_9f9ffbee4aaf", "status": "running", "progress": 0.4,
"detail": "Answer 1 of 2 · jurisdiction=US-TX — asking the Advisor", "run_ids": []}
When it is done (trimmed):
{"op_id": "op_9f9ffbee4aaf", "status": "done", "progress": 1.0, "run_ids": [1, 2], "error": null,
"result": {"question_id": "Q-rate-cap", "standing": 1.0, "run_ids": [1, 2],
"bindings": [{"answer_key_id": "dee-900", "binding_key": "jurisdiction=US-TX",
"run_id": 1, "standing": 1.0, "criteria_redacted": true}, "..."]}}
Poll every few seconds until status is done or error. Only the key that started the run sees
it. The op lives in the gateway's memory: after a gateway restart it is 404, but the stored runs
are still readable with GET /eval-runs.
8. Read the run¶
Trimmed:
{"id": 1, "question_id": "Q-rate-cap", "answer_key_id": "dee-900", "binding_key": "jurisdiction=US-TX",
"standing": 1.0, "cost_usd": 0.18468, "model": "together:moonshotai/Kimi-K3",
"judge_model": "together:openai/gpt-oss-120b", "brain_version": "edc4f008a431",
"ref": "draft", "seed": "0", "source": "live",
"results": [{"id": 1, "run_index": 0, "ok": true, "verdict": "PERMITTED", "share": 1.0,
"full_credit": true, "capped": false}]}
No route ever returns answer text. Per-criterion detail and penalty wording are returned only to a
key that may see rubrics. A result's id is what POST /eval-results/{id}/grades takes, if a
Curator grades it. List runs, newest first, with GET /eval-runs?question_id=Q-rate-cap.
9. Read the standing¶
{"brain_id": "meridian", "question_id": "Q-rate-cap",
"points": [{"at": "2026-11-07T15:31:45", "standing": 1.0, "run_ids": [1, 2],
"tuple": {"question_version": 1, "brain_version": "edc4f008a431",
"model": "together:moonshotai/Kimi-K3", "judge_model": "together:openai/gpt-oss-120b",
"seed": "0", "ref": "draft", "source": "live"}}]}
Points are oldest first. Two points compare only when their tuple matches; filter with
?model=…&ref=published and the other tuple fields.
10. Ask the Advisor, with question routing¶
The Advisor speaks OpenAI Chat Completions at /v1/brains/{brain_id}/chat/completions (scope
advisor:read). With advisor_question_routing on, it matches the ask to a published Question. If
the answer turns on facts the ask doesn't state (the Question's slots), it asks for them first.
curl -s -X POST "$B/chat/completions" -H "$AUTH" -H "Content-Type: application/json" \
-d '{"model": "polaris-advisor",
"messages": [{"role": "user", "content": "What notices do we owe when we decline on credit score and cash flow?"}]}'
The follow-up is an ordinary assistant message, so any OpenAI client just shows it. The structured
form rides beside choices as needs_input:
{"object": "chat.completion",
"choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant",
"content": "Before I answer, I need to know:\n- Is the credit consumer-purpose or business-purpose? (consumer, business)\n- Where does the linked-bank-account data come from? (a consumer reporting agency, a direct account link the applicant authorizes)\n- Are the score and cash-flow checks separate screens or one combined model? (separate screens, one combined model)\n\n(ref: a06a6f995470)"}}],
"needs_input": {"reason": "slots", "token": "1Xo1EfrASneiLIJBMMveZ6ownK25joxw",
"question_id": "Q-dee-48", "candidates": [],
"slots": [{"name": "credit_purpose", "type": "credit_purpose", "required": true,
"ask_text": "Is the credit consumer-purpose or business-purpose?",
"values": ["consumer", "business"]},
{"name": "bank_data_source", "...": "..."},
{"name": "decision_structure", "...": "..."}],
"message": ""}}
reason is slots (facts to fill) or choose_question (the ask could be one of several
Questions; candidates lists them). Resume it one of two ways.
As a plain OpenAI client. Send the history back with the person's reply as the next user turn.
Keep the assistant message as you got it: the (ref: …) at its end is how the gateway knows the next
turn is the reply.
curl -s -X POST "$B/chat/completions" -H "$AUTH" -H "Content-Type: application/json" \
-d '{"messages": [
{"role": "user", "content": "What notices do we owe when we decline on credit score and cash flow?"},
{"role": "assistant", "content": "Before I answer, I need to know: ...\n\n(ref: a06a6f995470)"},
{"role": "user", "content": "consumer credit, a direct account link, separate screens"}]}'
With a structured reply, when your app showed the choices. Send route_resume with the token
and either slots (the person's words per slot, a value or "I don't know") or question_id (for
choose_question; "none" answers unrouted):
curl -s -X POST "$B/chat/completions" -H "$AUTH" -H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "What notices do we owe when we decline on credit score and cash flow?"}],
"route_resume": {"token": "1Xo1EfrASneiLIJBMMveZ6ownK25joxw",
"slots": {"credit_purpose": "consumer",
"bank_data_source": "a direct account link the applicant authorizes",
"decision_structure": "separate screens"}}}'
The answer then comes back as usual, noting the facts it used:
<the answer>
This answer is based on:
- You told me: credit purpose: consumer; bank data source: a direct account link the applicant authorizes; decision structure: separate screens.
Sources:
- ...
If the reply leaves a required slot unfilled, you get another needs_input asking for what is still
missing. A reference is spent by its reply, and a pending route expires; after that the same history
is answered as a new question. A malformed route_resume is 400 with "param": "route_resume".
Over the Advisor MCP endpoint (/v1/mcp), the answer tool does the same: when it needs input,
structuredContent.needs_input is set and the text is the question to put to the person. Call
answer again with the same query, route_token set to needs_input.token, and slots,
question_id or reply. A reply without its route_token is refused (route_token_required).
See MCP.
11. The same flow over MCP¶
The remote QF MCP endpoint is /v1/mcp/questions/ (streamable HTTP), with the same key as a bearer
token. It serves the QF tools your key's scopes allow: list_questions, get_question,
question_standing, estimate_run, run_eval, propose_question, draft_answer_key,
request_approval, approve_answer_key and more. A key with no QF scope is refused at connect
(403 insufficient_scope).
from fastmcp import Client
from fastmcp.client.transports import StreamableHttpTransport
transport = StreamableHttpTransport(f"{POLARIS_URL}/v1/mcp/questions/",
headers={"Authorization": f"Bearer {POLARIS_KEY}"})
async with Client(transport) as mcp:
listed = (await mcp.call_tool("list_questions", {})).structured_content
q = (await mcp.call_tool("get_question", {"question_id": "Q-rate-cap"})).structured_content
list_questions returns {"brain_id", "count", "questions": [...]}; get_question returns
{"question", "rubric_visible", "answer_keys"}, with each Answer Key's rubric redacted for a Reader
key. A refusal is data, not an MCP error: {"code": "feature_disabled", "error": "...", "remedy": "..."}.
Check code on every result.
Run the quickstart¶
export POLARIS_URL=http://localhost:8088 POLARIS_BRAIN=meridian POLARIS_KEY=sk-polaris-...
export POLARIS_CURATOR_KEY=sk-polaris-... # or POLARIS_APPROVER=curator@example.com
export POLARIS_MAX_COST=0.50 # 0 skips the live eval
uv run python examples/qf_quickstart.py # outside the repo: pip install httpx fastmcp
It prints each step and what came back. A refused step prints the error code and a hint, and the
walk goes on where it can. Other settings: POLARIS_ASK (what to match and ask the Advisor),
POLARIS_REPLY (a free-text reply to a follow-up; without it, a structured route_resume) and
POLARIS_USE_DRAFTER=1 (draft the rubric with the drafter, a paid call).
Each run proposes a new draft Question, and with a Curator key approves and so publishes it. Point it at a test brain.
Troubleshooting¶
| Code | HTTP | What it means | What to do |
|---|---|---|---|
unauthorized |
401 | No key, a wrong one, or a revoked one. | Send Authorization: Bearer sk-polaris-…; mint a new key if it was revoked. |
feature_disabled |
403 | A QF flag is off for this brain. The message names it. | Ask a platform admin to turn it on (see Prerequisites). A key needs qf_developer_api and question_store_db; sign-off needs questions_workspace; versions and slots need question_slot_editing. |
insufficient_scope |
403 | The key lacks the route's scope. | Mint a key with it. Scopes are fixed at mint time. |
brain_forbidden |
403 | The key belongs to another brain. | Use a key minted for the brain in the path. |
role_forbidden |
403 | A Curator act (approve, sign off, publish a version, edit slots) with a Reader key. | Use a Curator or Admin key with the act's scope. |
approval_refused |
403 or 409 | detail.code says why: who is approving (403, for example not_a_person for an agent identity) or the draft's state (409). |
Approve as a person's key; fix the draft. |
citations_missing |
409 | With questions_workspace on, a criterion is uncited or cites a provision that doesn't resolve. |
Fix the citations in the Questions workspace, then approve. |
budget_exceeded |
402 | The brain's monthly eval budget would be exceeded. Checked before any model call. | Read GET /budget; ask an Admin to raise it, or wait for next month. |
max_cost_exceeded |
402 | The estimate is above your max_cost_usd. detail.estimate_usd has the estimate. |
Raise the ceiling, or ask for fewer runs or one binding. |
run_cap_exceeded |
400 | More runs than the brain allows per call. | Lower runs. |
seed_unpinned |
409 | The gateway was started without PYTHONHASHSEED, so a live run can't be compared or stored. |
Restart the gateway with PYTHONHASHSEED=0 (./scripts/dev.sh does this). |
idempotency_conflict |
409 | This Idempotency-Key was already used for a different request. |
Use a new key per distinct request; reuse a key only to retry the same one. |
idempotency_unavailable |
503 | The gateway can't store idempotency keys right now (no Redis). | Retry later, or send the request without the header. |
openrouter_key_required |
403 | /chat/completions: the brain has no OpenRouter key of its own, so it has no Advisor. The Advisor MCP reports it as a BLOCK with rule_id advisor.openrouter_key_required. |
A tenant Admin adds one under Settings > Model providers. |
not_found |
404 | No such Question, Answer Key or run, or an eval op that isn't yours or was lost in a restart. | Check the id; read stored runs with GET /eval-runs. |
/chat/completions uses OpenAI's error shape, {"error": {"message", "type", "code"}}, not the
QF one. Over MCP, a refusal is a result with code, error and remedy.