Skip to content

Walkthrough: the Questions Framework with an API key

This page takes you through the Questions Framework (QF) end to end, as an outside developer holding an API key. You list a brain's Questions, match an ask to one, propose a new Question, draft its Answer Key, get it approved, run a live eval, read the results and the standing, ask the Advisor a question that it routes to a Question, and do the same reads over MCP.

Every step is also a function in examples/qf_quickstart.py, in the same order. Run it to see the whole flow against your gateway (see Run the quickstart). A CI test runs the same functions against an in-process gateway, so the shapes below are the ones the code returns.

For the full reference, see:

Prerequisites

Flags a platform admin turns on

QF features are behind per-brain feature flags. Only a platform admin can change them, in the web app (Tenant settings > Feature flags) or with PUT /v1/brains/{brain_id}/flags/{flag_key}.

Flag Needed for
question_store_db Every QF route.
qf_developer_api Every QF route when the caller is an API key.
question_catalog GET /questions, POST /questions/match, GET /questions/{id}.
questions_workspace Certification sign-off and status. With it on, approval also requires every criterion to cite a provision that resolves.
question_slot_editing POST /questions/{id}/versions, PUT /questions/{id}/slots.
advisor_question_routing The Advisor matching an ask to a Question and asking for missing facts.

A platform admin sets a flag like this (with a platform admin's JWT):

curl -X PUT "$POLARIS_URL/v1/brains/$POLARIS_BRAIN/flags/qf_developer_api" \
  -H "Authorization: Bearer $ADMIN_JWT" -H "Content-Type: application/json" \
  -d '{"value": true}'

The Advisor also needs the brain's own OpenRouter key. A tenant Admin adds it in the web app under Settings > Model providers. Without it, /chat/completions returns 403 openrouter_key_required.

Mint the keys

A tenant Admin (or a platform admin) mints keys with their JWT. Get a JWT with POST /v1/auth/login:

ADMIN_JWT=$(curl -s -X POST "$POLARIS_URL/v1/auth/login" -H "Content-Type: application/json" \
  -d '{"email": "admin@example.com", "password": "..."}' | jq -r .access_token)

Mint a Reader key for your integration. It reads, proposes, drafts, runs evals and asks the Advisor:

curl -X POST "$POLARIS_URL/v1/brains/$POLARIS_BRAIN/api-keys" \
  -H "Authorization: Bearer $ADMIN_JWT" -H "Content-Type: application/json" \
  -d '{"label": "acme-integration", "role": "Reader",
       "scopes": ["questions:read", "questions:write", "eval:run", "advisor:read"]}'
{"id": 12, "brain_id": "meridian", "role": "Reader", "actor": "admin@example.com",
 "scopes": ["questions:read", "questions:write", "eval:run", "advisor:read"],
 "revoked": false, "key": "sk-polaris-..."}

The raw key is shown once. Keep it secret. Scopes are fixed when the key is minted; to change them, mint a new key and revoke the old one (DELETE /v1/brains/{brain_id}/api-keys/{key_id}).

If you also want to approve Answer Keys from code, mint a Curator key with answer_key:approve. Curator acts (answer_key:approve, certification:sign_off, questions:publish, slots:write) need a key with role Curator or Admin; minting refuses them on a Reader key. The key's actor (whoever minted it) is recorded as the approver, so a Curator should mint their own:

curl -X POST "$POLARIS_URL/v1/brains/$POLARIS_BRAIN/api-keys" \
  -H "Authorization: Bearer $CURATOR_OR_ADMIN_JWT" -H "Content-Type: application/json" \
  -d '{"label": "approvals", "role": "Curator", "scopes": ["questions:read", "answer_key:approve"]}'

A key is bound to its brain. Using it on another brain's path is 403 brain_forbidden.

Set up your shell

export POLARIS_URL=http://localhost:8088
export POLARIS_BRAIN=meridian
export POLARIS_KEY=sk-polaris-...            # the Reader key
export POLARIS_CURATOR_KEY=sk-polaris-...    # optional: the Curator key
B="$POLARIS_URL/v1/brains/$POLARIS_BRAIN"
AUTH="Authorization: Bearer $POLARIS_KEY"

Every refusal comes back as:

{"error": {"code": "feature_disabled", "message": "...", "detail": null}}

Branch on code; it is stable. See Troubleshooting.

1. List the published Questions

curl -s "$B/questions?limit=50" -H "$AUTH"
{"items": [
  {"question_id": "Q-rate-cap", "version": 1, "status": "published",
   "text": "What is the maximum rate on a consumer loan in {jurisdiction}?",
   "intent": "", "topic": "rates",
   "slots": [{"name": "jurisdiction", "type": "us_state", "required": true,
              "ask_text": null, "default": null}]}],
 "next_cursor": null}

Lists are paged. Pass next_cursor back as cursor until it is null. Narrow to one topic with ?topic=rates. Read one Question, with its Answer Keys, with GET /questions/{question_id}. A Reader key gets each Answer Key's rubric as null; only a Curator or Admin key with questions:read sees rubrics.

Next: match what a person asked to one of these.

2. Match an ask to a Question

curl -s -X POST "$B/questions/match" -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"question": "What is the maximum rate on a consumer loan?"}'
{"status": "matched", "question_id": "Q-rate-cap",
 "candidates": [{"question_id": "Q-rate-cap", "score": 1.0, "topic": "rates"}],
 "binding": {},
 "binding_problems": ["binding leaves slots unbound: ['jurisdiction']"],
 "scoped_to": []}

status is matched, uncertain or none. On uncertain, ask the person which candidate they meant; never bind. binding holds the slot values the ask states; check binding_problems before you use it. Matching is free: no model call.

Next: if no Question fits, propose one.

3. Propose a draft Question

Writes take an optional Idempotency-Key header. Send a fresh one per request; a retry with the same key returns the first response (with Idempotent-Replayed: true) and acts once.

curl -s -X POST "$B/questions/proposals" -H "$AUTH" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"text": "Is a late fee capped on a consumer loan?",
       "intent": "Whether state law limits the late fee a lender may charge",
       "topic": "fees", "slots": []}'

201:

{"question_id": "Q-is-a-late-fee-capped-on-870933", "version": 1, "is_draft": true,
 "store_commit": "3fc339a0...",
 "next_step": "draft its Answer Key: POST /questions/{question_id}/answer-keys/drafts; a Curator publishes the Question"}

The Question is a draft. Nobody is served or scored on it yet. A slotted Question names each slot in its text as {slot} and declares it in slots as name or name:type. No model call.

Next: draft the Answer Key that says what a right answer is.

4. Draft its Answer Key

curl -s -X POST "$B/questions/Q-is-a-late-fee-capped-on-870933/answer-keys/drafts" \
  -H "$AUTH" -H "Content-Type: application/json" -H "Idempotency-Key: $(uuidgen)" \
  -d '{"binding": {}, "answer": "PERMITTED within limits: the cap depends on the state statute.",
       "use_drafter": false}'

201:

{"answer_key_id": "is-a-late-fee-capped-on-870933", "version": 1, "agent_drafted": false,
 "store_commit": "bad4931c...", "estimate_usd": null, "rubric": null, "citations": null}
  • use_drafter: false (the default) writes a one-criterion skeleton of your expert answer. Free.
  • use_drafter: true has the drafter propose the rubric from answer: one model call, or two if you leave answer out and it writes the reference answer from the corpus. The calls are checked against the brain's monthly budget first (402 budget_exceeded) and estimate_usd says what they were priced at.
  • binding gives a value for each required slot. One Answer Key per binding.
  • rubric and citations are null unless your key may see rubrics (Curator or Admin with questions:read).

Next: a Curator approves the draft.

5. Get it approved

An Answer Key scores nothing until a Curator or Admin approves it. You have two ways.

Request approval from a named Curator (their login email). They decide in the web app:

curl -s -X POST "$B/approvals" -H "$AUTH" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"answer_key_id": "is-a-late-fee-capped-on-870933", "approver": "curator@example.com"}'
{"id": 1, "answer_key_id": "is-a-late-fee-capped-on-870933", "status": "pending",
 "requested_by": "admin@example.com", "approver": "curator@example.com",
 "created_at": "2026-11-07T15:31:48+00:00", "expires_at": "2026-11-21T15:31:48+00:00",
 "decided_by": null, "decided_at": null}

requested_by is the key's actor. List your requests with GET /approvals; withdraw one with POST /approvals/{id}/withdraw.

Or approve directly with a Curator key (answer_key:approve):

curl -s -X POST "$B/answer-keys/is-a-late-fee-capped-on-870933/approve" \
  -H "Authorization: Bearer $POLARIS_CURATOR_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" -d '{"note": "checked against the statute"}'
{"answer_key_id": "is-a-late-fee-capped-on-870933", "approved_by": "curator@example.com",
 "version": 1, "commit": "bb27f5a7...", "api_key_id": 13}

The key's actor is the approver, and the audit log records the key's id and your note. Approving runs the same checks as the web app; a draft that fails them is 409. An approval refused because of who is approving (an agent identity, a platform operator) is 403.

Approving the first Answer Key of a draft Question publishes that Question: it shows up in GET /questions from then on.

Next: measure how well the Advisor answers a Question.

6. Estimate and start a live eval

A live eval has the Advisor answer each binding of a Question runs times; Polaris judges, scores and stores every run. It costs money, so price it first. The estimate is free:

curl -s "$B/questions/Q-rate-cap/runs/estimate?runs=1&judge=true" -H "$AUTH"
{"estimate_usd": 0.36936, "remaining_usd": 49.63, "monthly_usd": 50.0, "allowed": true, "reason": null}

Start it with a ceiling, so it is refused if the estimate is higher (402 max_cost_exceeded):

curl -s -X POST "$B/questions/Q-rate-cap/runs" -H "$AUTH" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" -d '{"runs": 1, "judge": true, "max_cost_usd": 0.50}'

202:

{"op_id": "op_9f9ffbee4aaf", "status": "running", "estimate_usd": 0.36936, "run_ids": []}

ref is draft (the corpus as it stands) by default; only published runs can certify the published version. answer_key_id limits the run to one binding. The run is checked against the monthly budget (402 budget_exceeded) and the run cap (400 run_cap_exceeded) before any model call.

7. Poll the run

curl -s "$B/eval-ops/op_9f9ffbee4aaf" -H "$AUTH"

While it runs:

{"op_id": "op_9f9ffbee4aaf", "status": "running", "progress": 0.4,
 "detail": "Answer 1 of 2 · jurisdiction=US-TX — asking the Advisor", "run_ids": []}

When it is done (trimmed):

{"op_id": "op_9f9ffbee4aaf", "status": "done", "progress": 1.0, "run_ids": [1, 2], "error": null,
 "result": {"question_id": "Q-rate-cap", "standing": 1.0, "run_ids": [1, 2],
            "bindings": [{"answer_key_id": "dee-900", "binding_key": "jurisdiction=US-TX",
                          "run_id": 1, "standing": 1.0, "criteria_redacted": true}, "..."]}}

Poll every few seconds until status is done or error. Only the key that started the run sees it. The op lives in the gateway's memory: after a gateway restart it is 404, but the stored runs are still readable with GET /eval-runs.

8. Read the run

curl -s "$B/eval-runs/1" -H "$AUTH"

Trimmed:

{"id": 1, "question_id": "Q-rate-cap", "answer_key_id": "dee-900", "binding_key": "jurisdiction=US-TX",
 "standing": 1.0, "cost_usd": 0.18468, "model": "together:moonshotai/Kimi-K3",
 "judge_model": "together:openai/gpt-oss-120b", "brain_version": "edc4f008a431",
 "ref": "draft", "seed": "0", "source": "live",
 "results": [{"id": 1, "run_index": 0, "ok": true, "verdict": "PERMITTED", "share": 1.0,
              "full_credit": true, "capped": false}]}

No route ever returns answer text. Per-criterion detail and penalty wording are returned only to a key that may see rubrics. A result's id is what POST /eval-results/{id}/grades takes, if a Curator grades it. List runs, newest first, with GET /eval-runs?question_id=Q-rate-cap.

9. Read the standing

curl -s "$B/questions/Q-rate-cap/standing" -H "$AUTH"
{"brain_id": "meridian", "question_id": "Q-rate-cap",
 "points": [{"at": "2026-11-07T15:31:45", "standing": 1.0, "run_ids": [1, 2],
             "tuple": {"question_version": 1, "brain_version": "edc4f008a431",
                       "model": "together:moonshotai/Kimi-K3", "judge_model": "together:openai/gpt-oss-120b",
                       "seed": "0", "ref": "draft", "source": "live"}}]}

Points are oldest first. Two points compare only when their tuple matches; filter with ?model=…&ref=published and the other tuple fields.

10. Ask the Advisor, with question routing

The Advisor speaks OpenAI Chat Completions at /v1/brains/{brain_id}/chat/completions (scope advisor:read). With advisor_question_routing on, it matches the ask to a published Question. If the answer turns on facts the ask doesn't state (the Question's slots), it asks for them first.

curl -s -X POST "$B/chat/completions" -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"model": "polaris-advisor",
       "messages": [{"role": "user", "content": "What notices do we owe when we decline on credit score and cash flow?"}]}'

The follow-up is an ordinary assistant message, so any OpenAI client just shows it. The structured form rides beside choices as needs_input:

{"object": "chat.completion",
 "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant",
   "content": "Before I answer, I need to know:\n- Is the credit consumer-purpose or business-purpose? (consumer, business)\n- Where does the linked-bank-account data come from? (a consumer reporting agency, a direct account link the applicant authorizes)\n- Are the score and cash-flow checks separate screens or one combined model? (separate screens, one combined model)\n\n(ref: a06a6f995470)"}}],
 "needs_input": {"reason": "slots", "token": "1Xo1EfrASneiLIJBMMveZ6ownK25joxw",
   "question_id": "Q-dee-48", "candidates": [],
   "slots": [{"name": "credit_purpose", "type": "credit_purpose", "required": true,
              "ask_text": "Is the credit consumer-purpose or business-purpose?",
              "values": ["consumer", "business"]},
             {"name": "bank_data_source", "...": "..."},
             {"name": "decision_structure", "...": "..."}],
   "message": ""}}

reason is slots (facts to fill) or choose_question (the ask could be one of several Questions; candidates lists them). Resume it one of two ways.

As a plain OpenAI client. Send the history back with the person's reply as the next user turn. Keep the assistant message as you got it: the (ref: …) at its end is how the gateway knows the next turn is the reply.

curl -s -X POST "$B/chat/completions" -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"messages": [
        {"role": "user", "content": "What notices do we owe when we decline on credit score and cash flow?"},
        {"role": "assistant", "content": "Before I answer, I need to know: ...\n\n(ref: a06a6f995470)"},
        {"role": "user", "content": "consumer credit, a direct account link, separate screens"}]}'

With a structured reply, when your app showed the choices. Send route_resume with the token and either slots (the person's words per slot, a value or "I don't know") or question_id (for choose_question; "none" answers unrouted):

curl -s -X POST "$B/chat/completions" -H "$AUTH" -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "What notices do we owe when we decline on credit score and cash flow?"}],
       "route_resume": {"token": "1Xo1EfrASneiLIJBMMveZ6ownK25joxw",
                        "slots": {"credit_purpose": "consumer",
                                  "bank_data_source": "a direct account link the applicant authorizes",
                                  "decision_structure": "separate screens"}}}'

The answer then comes back as usual, noting the facts it used:

<the answer>

This answer is based on:
- You told me: credit purpose: consumer; bank data source: a direct account link the applicant authorizes; decision structure: separate screens.

Sources:
- ...

If the reply leaves a required slot unfilled, you get another needs_input asking for what is still missing. A reference is spent by its reply, and a pending route expires; after that the same history is answered as a new question. A malformed route_resume is 400 with "param": "route_resume".

Over the Advisor MCP endpoint (/v1/mcp), the answer tool does the same: when it needs input, structuredContent.needs_input is set and the text is the question to put to the person. Call answer again with the same query, route_token set to needs_input.token, and slots, question_id or reply. A reply without its route_token is refused (route_token_required). See MCP.

11. The same flow over MCP

The remote QF MCP endpoint is /v1/mcp/questions/ (streamable HTTP), with the same key as a bearer token. It serves the QF tools your key's scopes allow: list_questions, get_question, question_standing, estimate_run, run_eval, propose_question, draft_answer_key, request_approval, approve_answer_key and more. A key with no QF scope is refused at connect (403 insufficient_scope).

from fastmcp import Client
from fastmcp.client.transports import StreamableHttpTransport

transport = StreamableHttpTransport(f"{POLARIS_URL}/v1/mcp/questions/",
                                    headers={"Authorization": f"Bearer {POLARIS_KEY}"})
async with Client(transport) as mcp:
    listed = (await mcp.call_tool("list_questions", {})).structured_content
    q = (await mcp.call_tool("get_question", {"question_id": "Q-rate-cap"})).structured_content

list_questions returns {"brain_id", "count", "questions": [...]}; get_question returns {"question", "rubric_visible", "answer_keys"}, with each Answer Key's rubric redacted for a Reader key. A refusal is data, not an MCP error: {"code": "feature_disabled", "error": "...", "remedy": "..."}. Check code on every result.

Run the quickstart

export POLARIS_URL=http://localhost:8088 POLARIS_BRAIN=meridian POLARIS_KEY=sk-polaris-...
export POLARIS_CURATOR_KEY=sk-polaris-...      # or POLARIS_APPROVER=curator@example.com
export POLARIS_MAX_COST=0.50                   # 0 skips the live eval
uv run python examples/qf_quickstart.py        # outside the repo: pip install httpx fastmcp

It prints each step and what came back. A refused step prints the error code and a hint, and the walk goes on where it can. Other settings: POLARIS_ASK (what to match and ask the Advisor), POLARIS_REPLY (a free-text reply to a follow-up; without it, a structured route_resume) and POLARIS_USE_DRAFTER=1 (draft the rubric with the drafter, a paid call).

Each run proposes a new draft Question, and with a Curator key approves and so publishes it. Point it at a test brain.

Troubleshooting

Code HTTP What it means What to do
unauthorized 401 No key, a wrong one, or a revoked one. Send Authorization: Bearer sk-polaris-…; mint a new key if it was revoked.
feature_disabled 403 A QF flag is off for this brain. The message names it. Ask a platform admin to turn it on (see Prerequisites). A key needs qf_developer_api and question_store_db; sign-off needs questions_workspace; versions and slots need question_slot_editing.
insufficient_scope 403 The key lacks the route's scope. Mint a key with it. Scopes are fixed at mint time.
brain_forbidden 403 The key belongs to another brain. Use a key minted for the brain in the path.
role_forbidden 403 A Curator act (approve, sign off, publish a version, edit slots) with a Reader key. Use a Curator or Admin key with the act's scope.
approval_refused 403 or 409 detail.code says why: who is approving (403, for example not_a_person for an agent identity) or the draft's state (409). Approve as a person's key; fix the draft.
citations_missing 409 With questions_workspace on, a criterion is uncited or cites a provision that doesn't resolve. Fix the citations in the Questions workspace, then approve.
budget_exceeded 402 The brain's monthly eval budget would be exceeded. Checked before any model call. Read GET /budget; ask an Admin to raise it, or wait for next month.
max_cost_exceeded 402 The estimate is above your max_cost_usd. detail.estimate_usd has the estimate. Raise the ceiling, or ask for fewer runs or one binding.
run_cap_exceeded 400 More runs than the brain allows per call. Lower runs.
seed_unpinned 409 The gateway was started without PYTHONHASHSEED, so a live run can't be compared or stored. Restart the gateway with PYTHONHASHSEED=0 (./scripts/dev.sh does this).
idempotency_conflict 409 This Idempotency-Key was already used for a different request. Use a new key per distinct request; reuse a key only to retry the same one.
idempotency_unavailable 503 The gateway can't store idempotency keys right now (no Redis). Retry later, or send the request without the header.
openrouter_key_required 403 /chat/completions: the brain has no OpenRouter key of its own, so it has no Advisor. The Advisor MCP reports it as a BLOCK with rule_id advisor.openrouter_key_required. A tenant Admin adds one under Settings > Model providers.
not_found 404 No such Question, Answer Key or run, or an eval op that isn't yours or was lost in a restart. Check the id; read stored runs with GET /eval-runs.

/chat/completions uses OpenAI's error shape, {"error": {"message", "type", "code"}}, not the QF one. Over MCP, a refusal is a result with code, error and remedy.