Repository navigation
feat: send tool result media to providers like user files - #31383
Draft
ethanndickson wants to merge 9 commits into
Draft
ethanndickson wants to merge 9 commits into
ethanndickson wants to merge 9 commits into
Conversation
coder/fantasy#73 and coder/fantasy#74 convert tool result media through each provider's file part converter and add a filename. Repin to coder_2_33 once they merge. coder/fantasy#74 also stops sending text files as Chat Completions file parts, so Chat Completions routes no longer accept them and chatprompt inlines user text files as text, as it does on Responses.
… storage An MCP tool that returns an embedded resource blob now names the media after the last element of the resource URI path. chatloop copies the name into the media tool result, and chatprompt stores it in the persisted media result and restores it for the next prompt, so providers can label the file the same way they label a user attachment.
…transport as files Send a PDF tool result with a filename through the anthropic, OpenAI Chat Completions, OpenAI Responses and openai-compat clients and check the request body: an Anthropic document inside tool_result, a user file part right after the tool message on Chat Completions, and an input_file inside function_call_output on Responses, each with the filename.
… user file part chatd sends PDFs on Chat Completions as user file parts, both for attachments and for tool results. aibridge decodes the body into openai-go types and encodes it again, so check that file_data and filename survive.
…the path import resourceFilename imports the path package, and revive rejects the newServerRedactor loop variable named path.
…haracters A percent-encoded NUL in a resource URI decodes into the filename, and Postgres jsonb rejects \u0000, so the tool result could not be saved. Such a name is never a useful label, so send the media without one.
…atch the media type OpenAI detects the file type of an input file from its filename extension. A PDF from a URI such as https://api/x/files/12345 had no extension, and report.bin had a wrong one. The stored name is replayed on every turn, so a rejected name would break every later turn. Keep the name when its extension maps to the media type. Otherwise append the canonical extension, which is always .pdf for application/pdf. Return no name when the media type has no known extension, so the provider default applies.
The filename comes from the resource URI of an untrusted inline MCP server. A URI such as https://host/sse/<token> made the token the filename, which reached the model and the stored chat history. Redact the filename like the other tool response fields.
lint/go/new-code runs errcheck with check-blank, which rejects the discarded mime.ParseMediaType and mime.ExtensionsByType errors.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of CODAGT-1430. This is the first of four stacked PRs.
Today a PDF in an MCP tool result reaches no provider. On Anthropic we strip it in chatd. On OpenAI Chat Completions fantasy swaps it for a "cannot be displayed" note, and on Responses it sends a placeholder that points at a user message it never sends.
coder/fantasy#73 and coder/fantasy#74 make fantasy convert tool result media with the converter it already uses for user files: an Anthropic document inside
tool_result, aninput_fileinsidefunction_call_outputon Responses, and a user file part straight after the tool message on Chat Completions. This PR pointscharm.land/fantasyat coder/fantasy#74. Chat Completions file parts now refuse text files, as Responses already did, soAcceptsFilePartMediaTypefollows.The converters take a filename, so an MCP embedded resource blob is now named after the last element of its URI path, and we store the name with the tool result so later turns send the same one. OpenAI picks the file type from the extension, so we append the right one when it's missing or wrong. We drop names with control characters, as Postgres
jsonbrejects NUL, and we redact the name like the rest of the tool response, since a URI likehttps://host/sse/<token>would otherwise put the token in the chat history.There's also a test proving aibridge's Chat Completions re-encode keeps a user file part, as that's where these PDFs end up.
The
go.modreplace points at the coder/fantasy#74 head for now. Once both fantasy PRs merge I'll repin to the merged commit before this leaves draft. Anthropic still strips tool result PDFs until the next PR in the stack.