Skip to content

CodeTailor: call UMGPT when the course's token is for it - #1636

Merged
bnmnetp merged 9 commits into
RunestoneInteractive:mainfrom
aspadiyath:codetailor-umgpt
Nov 17, 2026
Merged

bnmnetp merged 9 commits into
RunestoneInteractive:mainfrom
aspadiyath:codetailor-umgpt

Conversation

@aspadiyath

@aspadiyath aspadiyath commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

CodeTailor only used a course's openai token and always called api.openai.com. It can now use a umgpt token too (entered through "Other" on the add-token page). CodeTailor and async peer instruction now share one setup for choosing the provider, model and client, instead of each keeping its own.

Shared LLM setup (components/rsptx/ai_helpers/api_token.py)

  • fetch_llm_token(course_id, feature) returns the course's key, its provider, and the feature using it. A course uses the provider it has a key for. If it has keys for several, the first in llm_provider_order wins (UMGPT, then OpenAI, the order PI already used).
  • The token supplies the model, the request parameters (request_params()), and a sync or async OpenAI client pointed at the provider's base URL.
  • masked() and repr() keep keys out of logs. Short keys show fewer characters.

Configuration (Settings in components/rsptx/configuration/core.py, each overridable by env var)

  • llm_provider_order, llm_base_url_<provider>, llm_model_<provider>. llm_model_umgpt defaults to PI_OPENAI_MODEL, so existing deployments keep their model.
  • llm_feature_models: per-feature overrides. PI on OpenAI stays on gpt-4o-mini, because it sends temperature, which GPT-5 models reject.
  • llm_model_params: extra parameters for each model (verbosity for gpt-5-nano), replacing the check on the model's name.

Peer instruction (assignment_server_api/routers/peer.py)

  • Drops LLM_BASE_URLS, _llm_provider_async and the two hand-written aiohttp calls. It uses the shared async client instead, with the same timeouts, no retries, and the client closed after each request.
  • The assignment server now bundles the ai_helpers brick and depends on openai.

CodeTailor

  • Both LLM calls take their client, model and parameters from the token.
  • Question loading and JOBE helpers moved to components/rsptx/code_runner/. The Java path now treats a failed upload or a non-JSON JOBE reply as a failing run (it used to count as passing).
  • The dev-key path no longer crashes before reaching the LLM.

Behavior in production

  • PI is unchanged on both providers.
  • CodeTailor on OpenAI stays on gpt-5-nano. On UMGPT it uses the same model as PI.
  • A course with both keys now gets UMGPT in CodeTailor, matching PI. In practice courses have one key.

Testing

  • Unit tests in test/components/rsptx/ai_helpers/ (provider choice, settings, request parameters, masking) and test/components/rsptx/code_runner/.
  • test/bases/rsptx/assignment_server_api/test_peer_llm.py runs the real OpenAI SDK with a mocked HTTP transport. It checks the URL, model and parameters PI sends on each provider, and that the client is closed.
  • test/bases/rsptx/book_server_api/test_java_jobe_failures.py covers the Java failure fix.
  • Tested by hand on localhost: CodeTailor Get Help with a UMGPT-only course returned an AI_personalized fix. Seth tested async PI on this branch.

🤖 Generated with Claude Code

aspadiyath and others added 4 commits October 26, 2026 13:38
No behavior change. Explain in Plain English questions (RunestoneInteractive#1627) need the
same question, token and JOBE plumbing CodeTailor has, so move it out of
coach.py and evaluate_fixed_code.py:

- activecode_questions.py: a question's starter code, tests and data
  files, plus find_question (base course first, then a global match) and
  question_test_code (question_json.suffix_code or the textarea tests)
- openai_token.py: the course's OpenAI token and the dev-course key
- jobe.py: the JOBE session, data file upload, and load_and_run_tests,
  now split around run_python_program, which returns a run's output

Adds tests for the moved helpers, which had none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodeTailor only looked up a course's "openai" token and always called
api.openai.com. Courses at U-M can now save a "umgpt" token instead,
and CodeTailor sends those requests to U-M's gateway, the same one
peer instruction uses.

The token lookup now returns the key with its provider; a table in
openai_token.py gives each provider's base URL and model. OpenAI is
tried first, so a course with both keeps the model it used. UMGPT uses
gpt-4o-mini, and verbosity is only sent to GPT-5 models.

Also adds UMGPT to the instructor add-token dropdown, and stops the
parsons_scaffolding debug print from writing the full key to stdout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instructors enter a UMGPT token through "Other" as "umgpt".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aspadiyath
aspadiyath requested a review from bnmnetp as a code owner October 28, 2026 15:51
@bnmnetp

bnmnetp commented Oct 28, 2026

Copy link
Copy Markdown
Member

I think the new files should be in components/rsptx/XXX

maybe we need a new component for llm_helpers. Not sure about the name so if you have better ideas.

…onents

The course LLM token lookup (openai_token.py, now api_token.py) goes in
ai_helpers. The question source/test loading and JOBE runner go in
code_runner, since they aren't AI-specific.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aspadiyath

Copy link
Copy Markdown
Contributor Author

Moved all three into components, split by purpose:

  • openai_token.py → components/rsptx/ai_helpers/api_token.py. I went with ai_helpers over llm_helpers so it can hold other AI code, and renamed the file since it handles UMGPT tokens too.
  • activecode_questions.py and jobe.py → components/rsptx/code_runner/. They aren't AI-specific (they load a question's tests/data files and run them on JOBE), so I put them in their own component.

Happy to rename either.

@bnmnetp bnmnetp left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Moving the token lookup and JOBE helpers into shared components is a good direction. Below are the issues I found, most important first. Line numbers are approximate.

Bugs (both existed before this PR, but they're in functions it touches)

  1. Failed Java test runs count as passing (evaluate_fixed_code.py ~L112). When a file upload to JOBE fails, or JOBE's reply isn't JSON, load_and_run_java_tests returns (False, {...}), and java_unittest_evaluation passes that tuple back as the pass/fail result. A non-empty tuple is truthy, so if unittest_result: treats untested LLM code as passing, and CodeTailor shows students an unverified "fixed" solution.
  2. course is never set on the dev path (coach.py ~L520). When uses_dev_api_key(course_name) is true, fetch_course never runs, so getattr(course, "base_course", None) raises UnboundLocalError. The except Exception turns that into a 400 "could not fetch test code", so the dev/test path never reaches the LLM.

Duplication and design

  1. A second provider table (components/rsptx/ai_helpers/api_token.py ~L20). LLM_PROVIDERS repeats LLM_BASE_URLS / _llm_provider_async in assignment_server_api/routers/peer.py, and the two now disagree. Peer instruction tries UMGPT first, while CodeTailor tries OpenAI first. Could peer.py use the new shared helper so there's only one copy?
  2. The UMGPT model is hardcoded (api_token.py ~L24). It's fixed at gpt-4o-mini and ignores PI_OPENAI_MODEL, which peer instruction uses for the same service. When that model is retired there, every CodeTailor call on UMGPT will fail.
  3. verbosity is chosen by model name (get_personalized_solution.py ~L168). Deciding with model.startswith("gpt-5") is a guess. Whether to send it would be better stored with each provider in LLM_PROVIDERS (e.g. an extra_params entry).
  4. The Java path still has its own JOBE code (evaluate_fixed_code.py ~L82). load_and_run_java_tests builds its own session, md5 file IDs and base64 encoding instead of using jobe.jobe_session(), jobe.runestone_file_id() and jobe.b64_text_utf8(). Any change to JOBE auth or file IDs would have to be made twice.
  5. Dead code was edited instead of deleted (evaluate_fixed_code.py ~L15). _ensure_file_on_jobe, PUSH_PROXY and CHECK_PROXY have no callers, so I'd remove them rather than update them.

Smaller

  1. Short keys aren't masked (api_token.py ~L54). masked() shows token[:4] and token[-4:], so an 8-character key prints in full and shorter ones leak completely, in both the logs and repr(). Please mask more for short tokens, or show only a fixed-length prefix.
  2. Two ways to load JOBE settings (code_runner/jobe.py ~L13). It imports settings from rsptx.configuration, while the Java path still uses from ..rsproxy import settings. Both resolve to the same object today, but they can drift apart.

The PR description is also out of date: it mentions openai_token.py and test/bases/.../test_openai_token.py, but the diff adds components/rsptx/ai_helpers/api_token.py, with tests under test/components/rsptx/ai_helpers/.

Suggested before merging: #1, #2 and #8, which are all small. Ideally also #3–4, so the two features can't drift apart on providers.

🤖 Generated with Claude Code

- A Java run whose file push fails or whose JOBE reply isn't JSON now
  returns False instead of a (False, info) tuple, which callers read as
  passing.
- Set course to None before the dev-key branch so the dev path can look
  up the question instead of raising UnboundLocalError.
- masked() shows fewer characters for short keys, so a short key is no
  longer printed in full.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aspadiyath

Copy link
Copy Markdown
Contributor Author

Fixed 1, 2 and 8:

  • 1: load_and_run_java_tests now returns plain False when a file push fails or JOBE's reply isn't JSON, so those runs count as failing. Added test/bases/rsptx/book_server_api/test_java_jobe_failures.py.
  • 2: course is set to None before the dev-key branch, so the dev/test path no longer crashes looking up the question.
  • 8: masked() now shows fewer characters for short keys (min(4, len // 6) at each end), so real keys keep their first/last four as a breadcrumb but a short key no longer prints in full.

I'll do 3 and 4 (one shared provider table for PI and CodeTailor, and the UMGPT model setting) as separate PRs after discussing with @sethbern, since they change PI's behavior too.

@aspadiyath

Copy link
Copy Markdown
Contributor Author

@bnmnetp Do you think this PR is ready for merge?

Barb and I would like to run an experiment that requires this PR in the next couple of weeks.

@bnmnetp

bnmnetp commented Nov 3, 2026

Copy link
Copy Markdown
Member

I was waiting for confirmation that you and @sethbern had a strategy to make model configuration consistent.

Lets try to get all that figured out and get this merged by the end of the week.

@aspadiyath

Copy link
Copy Markdown
Contributor Author

Ah I see, Seth is in Finland at a conference this week and weekend, so we won't be able to meet this week. But it's no worries, Barb and I have a backup plan for our study so there's no rush. Will let you know about the discussion with Seth next week after he comes back!

aspadiyath and others added 2 commits November 13, 2026 12:47
Both features now get the course's provider, model and client from
rsptx.ai_helpers.api_token, configured once in Settings:

- llm_provider_order, llm_base_url_<provider>, llm_model_<provider>:
  a course uses the provider it has a token for (UMGPT first if it has
  several, as PI already did). llm_model_umgpt defaults to
  PI_OPENAI_MODEL, so existing deployments keep their model.
- llm_feature_models: per-feature overrides. PI on OpenAI stays on
  gpt-4o-mini because it sends temperature, which GPT-5 models reject.
- llm_model_params: extra parameters for each model (verbosity for
  gpt-5-nano), replacing the model-name check.

peer.py drops its LLM_BASE_URLS table and hand-written aiohttp calls for
the shared AsyncOpenAI client (same timeouts, no retries). The assignment
server now bundles the ai_helpers brick and depends on openai.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	projects/assignment_server/pyproject.toml
@sethbern

Copy link
Copy Markdown
Contributor

I pulled it down and tested async PI with it, everything works for me. Looks good!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The changes leave incorrect provider selection for CodeTailor when both tokens exist and unclosed async HTTP clients that can retain sockets across reflection requests.

2 open issues
What changed in this PR

Adds UMGPT support to CodeTailor and shares LLM provider configuration with peer instruction. It also extracts reusable code-validation helpers into shared components.

Changes:

  • Centralizes provider selection, model settings, client creation, and API-key masking.
  • Moves activecode parsing and Python JOBE execution into shared helpers, and fixes Java failure results.
  • Adds focused regression tests and packages the shared components into the affected services.
File Description
test/​components/​rsptx/​code_runner/​test_clean_python_testcase.py Updates the helper import after extraction.
test/​components/​rsptx/​code_runner/​test_activecode_questions.py Tests source parsing, data-file discovery, and Python verdict handling.
test/​components/​rsptx/​code_runner/​__init__.py Adds the test package marker.
test/​components/​rsptx/​ai_helpers/​test_api_token.py Tests provider selection, model settings, and key masking.
test/​components/​rsptx/​ai_helpers/​__init__.py Adds the test package marker.
test/​bases/​rsptx/​book_server_api/​test_java_jobe_failures.py Tests Java upload and invalid-response failures.
test/​bases/​rsptx/​assignment_server_api/​test_peer_llm.py Tests peer-instruction requests through the shared client.
projects/​book_server/​pyproject.toml Packages both new shared components.
projects/​assignment_server/​pyproject.toml Adds the OpenAI dependency and AI helper component.
components/​rsptx/​configuration/​core.py Adds configurable provider, model, and request settings.
components/​rsptx/​code_runner/​jobe.py Extracts Python execution and data-file upload helpers.
components/​rsptx/​code_runner/​activecode_questions.py Extracts question lookup, parsing, and data-file resolution.
components/​rsptx/​code_runner/​__init__.py Exposes the code-runner modules.
components/​rsptx/​ai_helpers/​api_token.py Adds provider-aware tokens, clients, and masked representations.
components/​rsptx/​ai_helpers/​__init__.py Exposes the token helper module.
bases/​rsptx/​book_server_api/​routers/​personalized_parsons/​get_personalized_solution.py Routes both generation calls through provider-aware clients.
bases/​rsptx/​book_server_api/​routers/​personalized_parsons/​evaluate_fixed_code.py Reuses JOBE helpers and returns booleans for Java failures.
bases/​rsptx/​book_server_api/​routers/​personalized_parsons/​end_to_end.py Updates token parameter documentation.
bases/​rsptx/​book_server_api/​routers/​coach.py Uses shared helpers and logs provider details with masked keys.
bases/​rsptx/​assignment_server_api/​routers/​peer.py Consolidates LLM requests through the shared async client.

Review effort: Balanced

Comment thread components/rsptx/configuration/core.py Resolved
_llm_chat_async now opens the AsyncOpenAI client with `async with`, so its
connections are released on success and on errors instead of waiting for
garbage collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bnmnetp
bnmnetp merged commit 90a6e09 into RunestoneInteractive:main Nov 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants