Skip to content

Offer the tool pilot's calculator as a package and an MCP server - #269

Draft
MaxGhenis wants to merge 9 commits into
v2-microcosm-batchesfrom
calculator-mcp
Draft

MaxGhenis wants to merge 9 commits into
v2-microcosm-batchesfrom
calculator-mcp

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Makes the tool pilot's calculator usable outside the harness, as a package and as an MCP server, and prepares (does not run) an MCP arm. Stacked on #240. Draft: one piece of evidence is still owed (the replay on the engine, below), and the merge waits on the order with #257.

What it adds

  • policybench_calculator/core.py: a verbatim copy of the calculator inside scripts/tool_pilot_argon.py: the tool's schema and descriptions for each feature set, the validator (names, labels, placement, overrides, the flat household), the engine call, the explanations, the board's pinned engine and reform, and the worker pool. It is the dependency closure of 19 entry points: 102 module-level definitions, 2,013 lines. scripts/sync_calculator_core.py writes it from the harness (--check reports a stale copy).
  • policybench_calculator/mcp_server.py: an MCP server with one tool, calculate_household, built on the SDK's low-level Server (mcp 2.3.0), which sends the schema as written and validates nothing itself. It lists the name, description and input schema calculate_tool_schema builds for an arm, runs each call in the pilot's worker pool, answers with json.dumps of the result (the text the pilot's add_tool_result sends), refuses to start on an engine other than the board's, and offers the engine, board, features and schema hash as the resource policybench://calculator/provenance. Transports: stdio, or Streamable HTTP on 127.0.0.1 only, with no authentication. Run it with policybench-calculator-mcp --arm pe3.
  • policybench_calculator/mcp_client.py: McpToolCall, a tool_call for run_household (arguments in, result and seconds out) over an MCP session.
  • scripts/replay_calculator_calls.py: replays recorded calls through the harness's calculator and the package's, and with --mcp through the server. It writes comparisons and counts, never a call's arguments or result.
  • docs/calculator_mcp/design.md: the MCP arm's design and cost estimate.
  • pyproject.toml and uv.lock: an optional mcp extra (mcp==2.3.0, also in the dev extra) and the command above. The lock adds 13 packages and changes no other.

Why a copy and not a move

scripts/confirmatory_run.py refuses to start unless scripts/tool_pilot_argon.py, policybench/ and the fixes folder are its treatment commit's (TREATMENT_PATHS). So this PR changes nothing under those paths, and the package sits beside policybench/. After the confirmatory run, a follow-up makes the harness import the package and deletes its copy.

Checked on every commit: git diff --quiet c0a2885b <commit> -- scripts/tool_pilot_argon.py policybench reference_audit/2026-09-28/fixes exits 0.

Order with #257. The copy holds 102 of the harness's definitions, among them BASE_STATED_INPUTS and the v2-pilot wording tables, which the labels feature reads (rendered_labels calls v2_pilot_text). A PR that changes any of them in the harness makes test_core_is_the_sync_scripts_output_for_this_harness fail until python scripts/sync_calculator_core.py is run. Whichever of the two PRs lands second runs it, and the comparison is worth repeating at the study's freeze. Measured with the sync script's closure: the 102 definitions at this base equal treatment commit c0a2885's; the commits the development runs used (6b00a6f, 294e98f) differ only in V2_PILOT_CONVENTIONS (and FEATURES_BY_ARM at 6b00a6f).

Invariants the tests state

  1. core.py is exactly the sync script's output for the harness as it stands.
  2. Every module-level definition in core.py has the harness's syntax tree, in the harness's order (independent of how the script slices the source), and every constant has the harness's value.
  3. Both build the same tool schema, key order included, for all 96 feature sets (concise only beside explain) on two stated engines; and for 16 feature sets the schema hashes to the calculate_tool_sha256 the development runs recorded on policyengine-us 2.60.4.
  4. Both read any call alike (differential, Hypothesis, derandomized): 1,000 malformed messages and 2,000 written calls, framed and unframed. A companion test checks that the written calls reach every outcome (of 600: at least 150 accepted, 100 accepted with notes, 100 refused).
  5. Both run a request alike on a fake engine, and both pools send a worker the same message.
  6. The server lists the pilot's tool (name, description, input schema, no output schema) and returns the pilot's answer text for accepted, noted, explained and refused calls: in process, over stdio with a handshake-era session (2025-11-25), and over Streamable HTTP in both protocol eras (2025-11-25 and 2026-07-28). McpToolCall returns what ToolPool.call returns for arguments written as an object, as JSON text, and as text MCP cannot carry (answered with the validator's own refusal).
  7. A pool with no worker is a protocol error, never an answer a model would read.
  8. On the engine (slow test, the frozen board's policyengine-us 2.41.0): the server, started as its command starts it over stdio, gives the pilot's pool's answers for a household written for the test: accepted, read under a prompt label, explained in brief and in full, and refused.
  9. The replay script counts each recorded call once per distinct call, tells recorded timeouts and lost workers apart, resumes from its progress file, and exits 1 when the copies differ, when a call is left unreplayed, or (with --mcp) when a server answers differently.

Evidence

The workflow does not trigger for a PR into v2-microcosm-batches, so CI-only ci/pr269-* branches (the head plus one commit that edits the workflow's trigger or jobs) ran it:

  • Full suite on the head ca23666: run 44325709255, lint, app and test passed; 6,271 passed, 84 skipped, 49 deselected (the slow engine test is one of the deselected).
  • Head ca23666, the calculator's files with the slow test: run 44324820621: uv sync --locked, ruff, sync_calculator_core.py --check, 61 passed and 1 skipped (the skip is the replay of the non-public records), and the slow engine test passed (84 s; peak 4.0 GB).
  • The server's session tests ran on the real SDK for the first time in run 44315093560 and all passed.
  • uv.lock was resolved on a GitHub runner (uv 0.13.9): 13 packages added, none changed. Three of them (httpx2 and httpcore2 2.16.2, python-multipart 0.0.34) were released on 2026-11-16.
  • Owed: the replay on the development records. About 6,500 recorded calculate_household calls (20 households, every arm, policyengine-us 2.60.4) through both copies and, for the version 3 arms, through the server. The household set is not public, so the calls are not committed: tests/test_calculator_recorded_calls.py reads their folder from environment variables and skips without them. The build host is under a disk freeze, so this has not run; a guarded job starts it when the freeze allows, and its summary (counts only) will be cited here.

One thing found in the harness, not changed here

The pilot's version 1 pool sends a worker the model's arguments bare, and a bare None is the worker's stop signal: a call with no arguments at all ends that worker and returns "tool worker failed" (ToolPool.call, _worker_main). Pools with any feature frame their messages and are unaffected. tests/test_calculator_mcp.py records the behaviour; the fix belongs with the harness after the treatment pin lifts.

Not in this PR

No model call and no hosting. The paid arm through the providers' connectors waits for Max's decision (d3157); the design lists what a go would build (a token check, a per-household call log and call limit on the server, and a runner).

🤖 Generated with Claude Code

…ble package

policybench_calculator/core.py is a verbatim copy of the calculator inside
scripts/tool_pilot_argon.py (schema and descriptions per feature set, the
validator, the engine call, explanations, the board's pinned engine and
reform, the worker pool): the dependency closure of 19 entry points, 102
module-level definitions, written by scripts/sync_calculator_core.py.

It is a copy, not a move, because the confirmatory runner refuses to start
unless scripts/tool_pilot_argon.py and policybench/ are its treatment
commit's (confirmatory_run.py, TREATMENT_PATHS). Nothing under those paths
changes here.

tests/test_calculator_core.py holds the two copies together without the
engine: core.py is the sync script's output for the harness as it stands;
every definition has the harness's syntax tree; both build the same tool
schema for all 96 feature sets on two stated engines, and the schemas hash
to what the development runs recorded; both read 3,000 generated calls
alike (malformed ones, and written ones the validator accepts, notes or
refuses), run a request alike on a fake engine, and frame pool messages
alike.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 18, 2026 2:31pm UTC

Request Review

…ll shape

policybench_calculator/mcp_server.py serves one tool, calculate_household,
with the name, description and input schema the pilot gives a model for the
same arm, and answers a call with the text the pilot returns for it. It uses
the SDK's low-level Server, which sends the schema as written and validates
nothing itself, runs each call in the pilot's worker pool, refuses to start
on an engine other than the board's, and offers the engine, board, features
and schema hash as a resource. stdio, or Streamable HTTP on 127.0.0.1.

policybench_calculator/mcp_client.py is a tool_call (arguments to result and
seconds) over an MCP session, for the pilot's loop and for the tests.

Work in progress: written against mcp 2.3.0's source and docs but not yet
run against the SDK (a disk freeze on the build host holds the install), so
the tests that open a session skip until mcp is in the dev extra and the
lock. The 14 tests that need no session pass: the calculator offers the
pilot's tool and schema hash for five arms, and its answers are the pilot's
pool's for accepted, noted, explained and refused calls.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…he package's

scripts/replay_calculator_calls.py reads the tool pilot's household records
and, for every distinct calculate_household call, compares the two copies'
readings (worker_request, framed as the pool frames it) and their results on
one shared engine, compares the result with the one the run recorded
(counting recorded timeouts and lost workers apart), and reads the arguments
again as the JSON object an MCP client sends. With --mcp it also sends each
call through the package's MCP server. Any difference between the copies
exits 1. It appends one line per call, so a rerun resumes.

The development households are not public, so tests/
test_calculator_recorded_calls.py runs the real replay only when
POLICYBENCH_RECORDED_RUNS and POLICYBENCH_RECORDED_BOARD name the folders; its
other six tests check the script on records written in the test, with a fake
engine: the counts, the resume, the gate, and the exit code.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
docs/calculator_mcp/design.md separates two arms the word could mean. With
the harness as the MCP client the model sees the same prompt and tool, so a
paid run measures only what the free replay already does. With a provider's
connector as the client the provider shows the tool and runs the loop; that
is the arm designed here: what OpenAI, Anthropic and Google document (read
2026-11-18, each claim checked by a second reader), how a provider could
reach a local engine and what each route exposes, where the treatment can
stay the pilot's and where it cannot (the 8-turn and 4-call limits), how the
runner writes records that score and diagnose accept, and a cost estimate
from the development runs' own records ($1.58 at list for four models; a $10
cap asked). Design only: no model call, nothing hosted.

Also: the server answers an unexpected failure in its own words, since on a
handshake-era connection the SDK sends a client the exception's text.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
pyproject.toml gains an optional `mcp` extra (mcp==2.3.0, the latest release
on PyPI, uploaded 2026-10-02), puts the same pin in the dev extra so the
server's tests run wherever the suite does, and adds the command
policybench-calculator-mcp. uv.lock adds the 13 packages that pin brings and
changes no other (uv 0.13.9, resolved on a GitHub runner): mcp and mcp-types
2.3.0, httpx2 and httpcore2 2.16.2, httpx2-jsfetch 1.0, opentelemetry-api
1.45.2, pyjwt 2.15.1, python-multipart 0.0.34, pywin32 312 (Windows only),
sse-starlette 3.7.0, starlette 1.10.0, truststore 0.10.5 and uvicorn 0.54.0.
Three of them (httpx2, httpcore2, python-multipart) were released on
2026-11-16.

With the SDK installed, all seven session tests passed on their first run
(GitHub run 44315093560): the in-process client, stdio with a handshake-era
session, Streamable HTTP in both protocol eras, and McpToolCall in both
modes. They no longer skip when mcp is missing.

Also fixes the written-call generator in tests/test_calculator_core.py, which
raised when it drew "other shape" and then "people" for one call (the same
run found it; the two copies were never at fault).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The progress file no longer holds a call's result text, which carries values
from the non-public development households; a line keeps the call's hash and
the comparisons, as the module docstring now says.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The launch of the package's server for a feature set moves into
_server_parameters, so a test can put the fake-engine server in its place:
with --mcp every call the protocol can carry goes through a server, one per
feature set; refusals, which the validator alone decides, must match; and a
server whose engine answers differently fails the gate on every accepted
call. A second test pins the real launch command to the board and the call's
features.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A slow test starts the server as its command does (stdio, the frozen board,
the pilot's own worker with its output off the wire) and compares its answers
with the pilot's pool on the same board, for a household written for the
test: accepted, read under a label, explained in brief and in full, and
refused. It needs the frozen board's engine and two engine workers, so the
suite's default run leaves it out; run it with -m slow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…mparison

The test read same_over_mcp on every refused line, and the line for
arguments that are not JSON has none (GitHub run 44324015286; the path's own
counts had passed). It now checks the sendable lines and that the unsendable
one carries no comparison.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — ca23666f Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant