Skip to content

docs: Fix max_tokens placement and response schema on Public Endpoints model pages - #847

Open
promptless[bot] wants to merge 3 commits into
mainfrom
promptless/kimi-public-endpoint-max-tokens
Open

promptless[bot] wants to merge 3 commits into
mainfrom
promptless/kimi-public-endpoint-max-tokens

Conversation

@promptless

@promptless promptless Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Open in Promptless

The Moonshot Kimi and IBM Granite 4.0 Public Endpoints model pages placed max_tokens and the other sampling parameters inside a nested sampling_params object. On the native /run and /runsync route those nested values are silently ignored, so max_tokens has no effect and generation runs unbounded, driving up latency and cost with no error to signal the problem.

This corrects both pages so the sampling parameters sit at the top level of input, matching the convention the other text Public Endpoints already use. On the Moonshot Kimi page it also adds the reasoning_effort parameter (top-level, low/high/max, specific to kimi-k3), adds a note that the token cap counts reasoning tokens and that finish_reason ("length" vs "stop") confirms the cap took effect, and updates the response schema and example to the actual shape (an output array whose entries carry cost and a result chat completion, with usage reported as prompt_tokens/completion_tokens). The Granite 4.0 change is limited to the request side.

Trigger Events


Tip: Enable auto-create PR in your Configuration to review suggestions directly in GitHub 🤖

…l pages

On the native /run and /runsync route, sampling parameters must sit at the
top level of the input object; nesting them under sampling_params is silently
ignored. Flatten the request examples on the Moonshot Kimi and IBM Granite 4.0
pages, add the reasoning_effort parameter for kimi-k3, and correct the Moonshot
Kimi response schema to the actual output array / result shape.
<ParamField body="input.sampling_params.max_tokens" type="integer" default="2048">
Maximum number of tokens to generate.
<ParamField body="input.max_tokens" type="integer" default="2048">
Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`. This cap counts reasoning tokens as well as visible output tokens.

@promptless promptless Bot Sep 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

On the native /runsync route, place max_tokens at the top level of input. Nested sampling parameters are ignored.

View source

@mintlify

mintlify Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
runpod-docs 🟢 Ready View Preview Nov 12, 2026, 10:51 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

</ParamField>

<Note>
Sampling parameters such as `max_tokens` must sit at the top level of the `input` object on the native route. Nesting them inside a `sampling_params` object has no effect: the values are ignored, and generation continues until the model stops on its own. Check `finish_reason` in the response (`"length"` means the `max_tokens` cap was reached, `"stop"` means the model finished on its own) to confirm the cap took effect.

@promptless promptless Bot Sep 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

The native route reads sampling parameters from the top level of input. A finish_reason of length indicates the token cap was reached.

View source


<ParamField body="input.sampling_params.temperature" type="float" default="1">
<ParamField body="input.reasoning_effort" type="string" default="max">
Controls how much reasoning the model performs before answering. One of `low`, `high`, or `max`. Specific to the `kimi-k3` variant.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Moonshot AI's own docs confirm reasoning_effort accepts low/high/max (default max) and is specific to Kimi K3 as the top-level replacement for the K2.x thinking configuration.

View source


<ParamField body="input.sampling_params.temperature" type="float" default="1">
<ParamField body="input.reasoning_effort" type="string" default="max">
Controls how much reasoning the model performs before answering. One of `low`, `high`, or `max`. Specific to the `kimi-k3` variant.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Runpod's Kimi K3 technical FAQ corroborates reasoning_effort values low/high/max for the K3 variant served on this endpoint.

View source

{
"cost": 0.0030309,
"result": {
"id": "chatcmpl-6aaa2dce25da6d26c138da0d",

@promptless promptless Bot Sep 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

The response contains an output array with per-result cost and a chat completion under result. Usage includes prompt, completion, and reasoning token counts.

View source

<ParamField body="input.sampling_params.max_tokens" type="integer" default="512">
Maximum number of tokens to generate.
<ParamField body="input.max_tokens" type="integer" default="512">
Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`.

@promptless promptless Bot Sep 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

The Granite 4 change applies the shared native-route input shape. Verification covered Kimi.

View source

<ParamField body="input.sampling_params.max_tokens" type="integer" default="512">
Maximum number of tokens to generate.
<ParamField body="input.max_tokens" type="integer" default="512">
Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Citation

Internal corroboration: sibling Public Endpoints model pages (Cogito 671B, GPT-OSS 120B, Qwen3 32B) already document max_tokens/temperature/top_p/top_k at the top level of input on the native route, with no sampling_params wrapper, consistent with the corrected placement.

Source: cogito-671b.mdx:31 · 1ac8c64

This branch was successfully deployed

1 active deployment
staging — 4836559d Deployed Nov 12, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant