Repository navigation
Conversation
…ble package policybench_calculator/core.py is a verbatim copy of the calculator inside scripts/tool_pilot_argon.py (schema and descriptions per feature set, the validator, the engine call, explanations, the board's pinned engine and reform, the worker pool): the dependency closure of 19 entry points, 102 module-level definitions, written by scripts/sync_calculator_core.py. It is a copy, not a move, because the confirmatory runner refuses to start unless scripts/tool_pilot_argon.py and policybench/ are its treatment commit's (confirmatory_run.py, TREATMENT_PATHS). Nothing under those paths changes here. tests/test_calculator_core.py holds the two copies together without the engine: core.py is the sync script's output for the harness as it stands; every definition has the harness's syntax tree; both build the same tool schema for all 96 feature sets on two stated engines, and the schemas hash to what the development runs recorded; both read 3,000 generated calls alike (malformed ones, and written ones the validator accepts, notes or refuses), run a request alike on a fake engine, and frame pool messages alike. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…ll shape policybench_calculator/mcp_server.py serves one tool, calculate_household, with the name, description and input schema the pilot gives a model for the same arm, and answers a call with the text the pilot returns for it. It uses the SDK's low-level Server, which sends the schema as written and validates nothing itself, runs each call in the pilot's worker pool, refuses to start on an engine other than the board's, and offers the engine, board, features and schema hash as a resource. stdio, or Streamable HTTP on 127.0.0.1. policybench_calculator/mcp_client.py is a tool_call (arguments to result and seconds) over an MCP session, for the pilot's loop and for the tests. Work in progress: written against mcp 2.3.0's source and docs but not yet run against the SDK (a disk freeze on the build host holds the install), so the tests that open a session skip until mcp is in the dev extra and the lock. The 14 tests that need no session pass: the calculator offers the pilot's tool and schema hash for five arms, and its answers are the pilot's pool's for accepted, noted, explained and refused calls. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…he package's scripts/replay_calculator_calls.py reads the tool pilot's household records and, for every distinct calculate_household call, compares the two copies' readings (worker_request, framed as the pool frames it) and their results on one shared engine, compares the result with the one the run recorded (counting recorded timeouts and lost workers apart), and reads the arguments again as the JSON object an MCP client sends. With --mcp it also sends each call through the package's MCP server. Any difference between the copies exits 1. It appends one line per call, so a rerun resumes. The development households are not public, so tests/ test_calculator_recorded_calls.py runs the real replay only when POLICYBENCH_RECORDED_RUNS and POLICYBENCH_RECORDED_BOARD name the folders; its other six tests check the script on records written in the test, with a fake engine: the counts, the resume, the gate, and the exit code. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
docs/calculator_mcp/design.md separates two arms the word could mean. With the harness as the MCP client the model sees the same prompt and tool, so a paid run measures only what the free replay already does. With a provider's connector as the client the provider shows the tool and runs the loop; that is the arm designed here: what OpenAI, Anthropic and Google document (read 2026-11-18, each claim checked by a second reader), how a provider could reach a local engine and what each route exposes, where the treatment can stay the pilot's and where it cannot (the 8-turn and 4-call limits), how the runner writes records that score and diagnose accept, and a cost estimate from the development runs' own records ($1.58 at list for four models; a $10 cap asked). Design only: no model call, nothing hosted. Also: the server answers an unexpected failure in its own words, since on a handshake-era connection the SDK sends a client the exception's text. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
pyproject.toml gains an optional `mcp` extra (mcp==2.3.0, the latest release on PyPI, uploaded 2026-10-02), puts the same pin in the dev extra so the server's tests run wherever the suite does, and adds the command policybench-calculator-mcp. uv.lock adds the 13 packages that pin brings and changes no other (uv 0.13.9, resolved on a GitHub runner): mcp and mcp-types 2.3.0, httpx2 and httpcore2 2.16.2, httpx2-jsfetch 1.0, opentelemetry-api 1.45.2, pyjwt 2.15.1, python-multipart 0.0.34, pywin32 312 (Windows only), sse-starlette 3.7.0, starlette 1.10.0, truststore 0.10.5 and uvicorn 0.54.0. Three of them (httpx2, httpcore2, python-multipart) were released on 2026-11-16. With the SDK installed, all seven session tests passed on their first run (GitHub run 44315093560): the in-process client, stdio with a handshake-era session, Streamable HTTP in both protocol eras, and McpToolCall in both modes. They no longer skip when mcp is missing. Also fixes the written-call generator in tests/test_calculator_core.py, which raised when it drew "other shape" and then "people" for one call (the same run found it; the two copies were never at fault). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The progress file no longer holds a call's result text, which carries values from the non-public development households; a line keeps the call's hash and the comparisons, as the module docstring now says. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The launch of the package's server for a feature set moves into _server_parameters, so a test can put the fake-engine server in its place: with --mcp every call the protocol can carry goes through a server, one per feature set; refusals, which the validator alone decides, must match; and a server whose engine answers differently fails the gate on every accepted call. A second test pins the real launch command to the board and the call's features. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A slow test starts the server as its command does (stdio, the frozen board, the pilot's own worker with its output off the wire) and compares its answers with the pilot's pool on the same board, for a household written for the test: accepted, read under a label, explained in brief and in full, and refused. It needs the frozen board's engine and two engine workers, so the suite's default run leaves it out; run it with -m slow. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…mparison The test read same_over_mcp on every refused line, and the line for arguments that are not JSON has none (GitHub run 44324015286; the path's own counts had passed). It now checks the sendable lines and that the unsendable one carries no comparison. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the tool pilot's calculator usable outside the harness, as a package and as an MCP server, and prepares (does not run) an MCP arm. Stacked on #240. Draft: one piece of evidence is still owed (the replay on the engine, below), and the merge waits on the order with #257.
What it adds
policybench_calculator/core.py: a verbatim copy of the calculator insidescripts/tool_pilot_argon.py: the tool's schema and descriptions for each feature set, the validator (names, labels, placement, overrides, the flat household), the engine call, the explanations, the board's pinned engine and reform, and the worker pool. It is the dependency closure of 19 entry points: 102 module-level definitions, 2,013 lines.scripts/sync_calculator_core.pywrites it from the harness (--checkreports a stale copy).policybench_calculator/mcp_server.py: an MCP server with one tool,calculate_household, built on the SDK's low-levelServer(mcp 2.3.0), which sends the schema as written and validates nothing itself. It lists the name, description and input schemacalculate_tool_schemabuilds for an arm, runs each call in the pilot's worker pool, answers withjson.dumpsof the result (the text the pilot'sadd_tool_resultsends), refuses to start on an engine other than the board's, and offers the engine, board, features and schema hash as the resourcepolicybench://calculator/provenance. Transports: stdio, or Streamable HTTP on 127.0.0.1 only, with no authentication. Run it withpolicybench-calculator-mcp --arm pe3.policybench_calculator/mcp_client.py:McpToolCall, atool_callforrun_household(arguments in, result and seconds out) over an MCP session.scripts/replay_calculator_calls.py: replays recorded calls through the harness's calculator and the package's, and with--mcpthrough the server. It writes comparisons and counts, never a call's arguments or result.docs/calculator_mcp/design.md: the MCP arm's design and cost estimate.pyproject.tomlanduv.lock: an optionalmcpextra (mcp==2.3.0, also in the dev extra) and the command above. The lock adds 13 packages and changes no other.Why a copy and not a move
scripts/confirmatory_run.pyrefuses to start unlessscripts/tool_pilot_argon.py,policybench/and the fixes folder are its treatment commit's (TREATMENT_PATHS). So this PR changes nothing under those paths, and the package sits besidepolicybench/. After the confirmatory run, a follow-up makes the harness import the package and deletes its copy.Checked on every commit:
git diff --quiet c0a2885b <commit> -- scripts/tool_pilot_argon.py policybench reference_audit/2026-09-28/fixesexits 0.Order with #257. The copy holds 102 of the harness's definitions, among them
BASE_STATED_INPUTSand the v2-pilot wording tables, which the labels feature reads (rendered_labelscallsv2_pilot_text). A PR that changes any of them in the harness makestest_core_is_the_sync_scripts_output_for_this_harnessfail untilpython scripts/sync_calculator_core.pyis run. Whichever of the two PRs lands second runs it, and the comparison is worth repeating at the study's freeze. Measured with the sync script's closure: the 102 definitions at this base equal treatment commit c0a2885's; the commits the development runs used (6b00a6f, 294e98f) differ only inV2_PILOT_CONVENTIONS(andFEATURES_BY_ARMat 6b00a6f).Invariants the tests state
core.pyis exactly the sync script's output for the harness as it stands.core.pyhas the harness's syntax tree, in the harness's order (independent of how the script slices the source), and every constant has the harness's value.calculate_tool_sha256the development runs recorded on policyengine-us 2.60.4.McpToolCallreturns whatToolPool.callreturns for arguments written as an object, as JSON text, and as text MCP cannot carry (answered with the validator's own refusal).--mcp) when a server answers differently.Evidence
The workflow does not trigger for a PR into
v2-microcosm-batches, so CI-onlyci/pr269-*branches (the head plus one commit that edits the workflow's trigger or jobs) ran it:uv sync --locked, ruff,sync_calculator_core.py --check, 61 passed and 1 skipped (the skip is the replay of the non-public records), and the slow engine test passed (84 s; peak 4.0 GB).uv.lockwas resolved on a GitHub runner (uv 0.13.9): 13 packages added, none changed. Three of them (httpx2 and httpcore2 2.16.2, python-multipart 0.0.34) were released on 2026-11-16.calculate_householdcalls (20 households, every arm, policyengine-us 2.60.4) through both copies and, for the version 3 arms, through the server. The household set is not public, so the calls are not committed:tests/test_calculator_recorded_calls.pyreads their folder from environment variables and skips without them. The build host is under a disk freeze, so this has not run; a guarded job starts it when the freeze allows, and its summary (counts only) will be cited here.One thing found in the harness, not changed here
The pilot's version 1 pool sends a worker the model's arguments bare, and a bare
Noneis the worker's stop signal: a call with no arguments at all ends that worker and returns "tool worker failed" (ToolPool.call,_worker_main). Pools with any feature frame their messages and are unaffected.tests/test_calculator_mcp.pyrecords the behaviour; the fix belongs with the harness after the treatment pin lifts.Not in this PR
No model call and no hosting. The paid arm through the providers' connectors waits for Max's decision (d3157); the design lists what a go would build (a token check, a per-household call log and call limit on the server, and a runner).
🤖 Generated with Claude Code