Repository navigation
CodeTailor: call UMGPT when the course's token is for it - #1636
Conversation
No behavior change. Explain in Plain English questions (RunestoneInteractive#1627) need the same question, token and JOBE plumbing CodeTailor has, so move it out of coach.py and evaluate_fixed_code.py: - activecode_questions.py: a question's starter code, tests and data files, plus find_question (base course first, then a global match) and question_test_code (question_json.suffix_code or the textarea tests) - openai_token.py: the course's OpenAI token and the dev-course key - jobe.py: the JOBE session, data file upload, and load_and_run_tests, now split around run_python_program, which returns a run's output Adds tests for the moved helpers, which had none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeTailor only looked up a course's "openai" token and always called api.openai.com. Courses at U-M can now save a "umgpt" token instead, and CodeTailor sends those requests to U-M's gateway, the same one peer instruction uses. The token lookup now returns the key with its provider; a table in openai_token.py gives each provider's base URL and model. OpenAI is tried first, so a course with both keeps the model it used. UMGPT uses gpt-4o-mini, and verbosity is only sent to GPT-5 models. Also adds UMGPT to the instructor add-token dropdown, and stops the parsons_scaffolding debug print from writing the full key to stdout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instructors enter a UMGPT token through "Other" as "umgpt". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I think the new files should be in components/rsptx/XXX maybe we need a new component for |
…onents The course LLM token lookup (openai_token.py, now api_token.py) goes in ai_helpers. The question source/test loading and JOBE runner go in code_runner, since they aren't AI-specific. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Moved all three into components, split by purpose:
Happy to rename either. |
bnmnetp
left a comment
There was a problem hiding this comment.
Thanks for this. Moving the token lookup and JOBE helpers into shared components is a good direction. Below are the issues I found, most important first. Line numbers are approximate.
Bugs (both existed before this PR, but they're in functions it touches)
- Failed Java test runs count as passing (
evaluate_fixed_code.py~L112). When a file upload to JOBE fails, or JOBE's reply isn't JSON,load_and_run_java_testsreturns(False, {...}), andjava_unittest_evaluationpasses that tuple back as the pass/fail result. A non-empty tuple is truthy, soif unittest_result:treats untested LLM code as passing, and CodeTailor shows students an unverified "fixed" solution. courseis never set on the dev path (coach.py~L520). Whenuses_dev_api_key(course_name)is true,fetch_coursenever runs, sogetattr(course, "base_course", None)raisesUnboundLocalError. Theexcept Exceptionturns that into a 400 "could not fetch test code", so the dev/test path never reaches the LLM.
Duplication and design
- A second provider table (
components/rsptx/ai_helpers/api_token.py~L20).LLM_PROVIDERSrepeatsLLM_BASE_URLS/_llm_provider_asyncinassignment_server_api/routers/peer.py, and the two now disagree. Peer instruction tries UMGPT first, while CodeTailor tries OpenAI first. Couldpeer.pyuse the new shared helper so there's only one copy? - The UMGPT model is hardcoded (
api_token.py~L24). It's fixed atgpt-4o-miniand ignoresPI_OPENAI_MODEL, which peer instruction uses for the same service. When that model is retired there, every CodeTailor call on UMGPT will fail. verbosityis chosen by model name (get_personalized_solution.py~L168). Deciding withmodel.startswith("gpt-5")is a guess. Whether to send it would be better stored with each provider inLLM_PROVIDERS(e.g. anextra_paramsentry).- The Java path still has its own JOBE code (
evaluate_fixed_code.py~L82).load_and_run_java_testsbuilds its own session, md5 file IDs and base64 encoding instead of usingjobe.jobe_session(),jobe.runestone_file_id()andjobe.b64_text_utf8(). Any change to JOBE auth or file IDs would have to be made twice. - Dead code was edited instead of deleted (
evaluate_fixed_code.py~L15)._ensure_file_on_jobe,PUSH_PROXYandCHECK_PROXYhave no callers, so I'd remove them rather than update them.
Smaller
- Short keys aren't masked (
api_token.py~L54).masked()showstoken[:4]andtoken[-4:], so an 8-character key prints in full and shorter ones leak completely, in both the logs andrepr(). Please mask more for short tokens, or show only a fixed-length prefix. - Two ways to load JOBE settings (
code_runner/jobe.py~L13). It imports settings fromrsptx.configuration, while the Java path still usesfrom ..rsproxy import settings. Both resolve to the same object today, but they can drift apart.
The PR description is also out of date: it mentions openai_token.py and test/bases/.../test_openai_token.py, but the diff adds components/rsptx/ai_helpers/api_token.py, with tests under test/components/rsptx/ai_helpers/.
Suggested before merging: #1, #2 and #8, which are all small. Ideally also #3–4, so the two features can't drift apart on providers.
🤖 Generated with Claude Code
- A Java run whose file push fails or whose JOBE reply isn't JSON now returns False instead of a (False, info) tuple, which callers read as passing. - Set course to None before the dev-key branch so the dev path can look up the question instead of raising UnboundLocalError. - masked() shows fewer characters for short keys, so a short key is no longer printed in full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fixed 1, 2 and 8:
I'll do 3 and 4 (one shared provider table for PI and CodeTailor, and the UMGPT model setting) as separate PRs after discussing with @sethbern, since they change PI's behavior too. |
|
@bnmnetp Do you think this PR is ready for merge? Barb and I would like to run an experiment that requires this PR in the next couple of weeks. |
|
I was waiting for confirmation that you and @sethbern had a strategy to make model configuration consistent. Lets try to get all that figured out and get this merged by the end of the week. |
|
Ah I see, Seth is in Finland at a conference this week and weekend, so we won't be able to meet this week. But it's no worries, Barb and I have a backup plan for our study so there's no rush. Will let you know about the discussion with Seth next week after he comes back! |
Both features now get the course's provider, model and client from rsptx.ai_helpers.api_token, configured once in Settings: - llm_provider_order, llm_base_url_<provider>, llm_model_<provider>: a course uses the provider it has a token for (UMGPT first if it has several, as PI already did). llm_model_umgpt defaults to PI_OPENAI_MODEL, so existing deployments keep their model. - llm_feature_models: per-feature overrides. PI on OpenAI stays on gpt-4o-mini because it sends temperature, which GPT-5 models reject. - llm_model_params: extra parameters for each model (verbosity for gpt-5-nano), replacing the model-name check. peer.py drops its LLM_BASE_URLS table and hand-written aiohttp calls for the shared AsyncOpenAI client (same timeouts, no retries). The assignment server now bundles the ai_helpers brick and depends on openai. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # projects/assignment_server/pyproject.toml
|
I pulled it down and tested async PI with it, everything works for me. Looks good! |
There was a problem hiding this comment.
🟡 Changes recommended
The changes leave incorrect provider selection for CodeTailor when both tokens exist and unclosed async HTTP clients that can retain sockets across reflection requests.
2 open issues
What changed in this PR
Adds UMGPT support to CodeTailor and shares LLM provider configuration with peer instruction. It also extracts reusable code-validation helpers into shared components.
Changes:
- Centralizes provider selection, model settings, client creation, and API-key masking.
- Moves activecode parsing and Python JOBE execution into shared helpers, and fixes Java failure results.
- Adds focused regression tests and packages the shared components into the affected services.
| File | Description |
|---|---|
test/components/rsptx/code_runner/test_clean_python_testcase.py |
Updates the helper import after extraction. |
test/components/rsptx/code_runner/test_activecode_questions.py |
Tests source parsing, data-file discovery, and Python verdict handling. |
test/components/rsptx/code_runner/__init__.py |
Adds the test package marker. |
test/components/rsptx/ai_helpers/test_api_token.py |
Tests provider selection, model settings, and key masking. |
test/components/rsptx/ai_helpers/__init__.py |
Adds the test package marker. |
test/bases/rsptx/book_server_api/test_java_jobe_failures.py |
Tests Java upload and invalid-response failures. |
test/bases/rsptx/assignment_server_api/test_peer_llm.py |
Tests peer-instruction requests through the shared client. |
projects/book_server/pyproject.toml |
Packages both new shared components. |
projects/assignment_server/pyproject.toml |
Adds the OpenAI dependency and AI helper component. |
components/rsptx/configuration/core.py |
Adds configurable provider, model, and request settings. |
components/rsptx/code_runner/jobe.py |
Extracts Python execution and data-file upload helpers. |
components/rsptx/code_runner/activecode_questions.py |
Extracts question lookup, parsing, and data-file resolution. |
components/rsptx/code_runner/__init__.py |
Exposes the code-runner modules. |
components/rsptx/ai_helpers/api_token.py |
Adds provider-aware tokens, clients, and masked representations. |
components/rsptx/ai_helpers/__init__.py |
Exposes the token helper module. |
bases/rsptx/book_server_api/routers/personalized_parsons/get_personalized_solution.py |
Routes both generation calls through provider-aware clients. |
bases/rsptx/book_server_api/routers/personalized_parsons/evaluate_fixed_code.py |
Reuses JOBE helpers and returns booleans for Java failures. |
bases/rsptx/book_server_api/routers/personalized_parsons/end_to_end.py |
Updates token parameter documentation. |
bases/rsptx/book_server_api/routers/coach.py |
Uses shared helpers and logs provider details with masked keys. |
bases/rsptx/assignment_server_api/routers/peer.py |
Consolidates LLM requests through the shared async client. |
Review effort: Balanced
_llm_chat_async now opens the AsyncOpenAI client with `async with`, so its connections are released on success and on errors instead of waiting for garbage collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CodeTailor only used a course's
openaitoken and always called api.openai.com. It can now use aumgpttoken too (entered through "Other" on the add-token page). CodeTailor and async peer instruction now share one setup for choosing the provider, model and client, instead of each keeping its own.Shared LLM setup (
components/rsptx/ai_helpers/api_token.py)fetch_llm_token(course_id, feature)returns the course's key, its provider, and the feature using it. A course uses the provider it has a key for. If it has keys for several, the first inllm_provider_orderwins (UMGPT, then OpenAI, the order PI already used).request_params()), and a sync or async OpenAI client pointed at the provider's base URL.masked()andrepr()keep keys out of logs. Short keys show fewer characters.Configuration (
Settingsincomponents/rsptx/configuration/core.py, each overridable by env var)llm_provider_order,llm_base_url_<provider>,llm_model_<provider>.llm_model_umgptdefaults toPI_OPENAI_MODEL, so existing deployments keep their model.llm_feature_models: per-feature overrides. PI on OpenAI stays ongpt-4o-mini, because it sendstemperature, which GPT-5 models reject.llm_model_params: extra parameters for each model (verbosityforgpt-5-nano), replacing the check on the model's name.Peer instruction (
assignment_server_api/routers/peer.py)LLM_BASE_URLS,_llm_provider_asyncand the two hand-writtenaiohttpcalls. It uses the shared async client instead, with the same timeouts, no retries, and the client closed after each request.ai_helpersbrick and depends onopenai.CodeTailor
components/rsptx/code_runner/. The Java path now treats a failed upload or a non-JSON JOBE reply as a failing run (it used to count as passing).Behavior in production
gpt-5-nano. On UMGPT it uses the same model as PI.Testing
test/components/rsptx/ai_helpers/(provider choice, settings, request parameters, masking) andtest/components/rsptx/code_runner/.test/bases/rsptx/assignment_server_api/test_peer_llm.pyruns the real OpenAI SDK with a mocked HTTP transport. It checks the URL, model and parameters PI sends on each provider, and that the client is closed.test/bases/rsptx/book_server_api/test_java_jobe_failures.pycovers the Java failure fix.AI_personalizedfix. Seth tested async PI on this branch.🤖 Generated with Claude Code