Skip to content

feat(openai): support output logprobs for chat completions - #1701

Open
WANDY666 wants to merge 3 commits into
mainfrom
openai-chat-output-logprobs
Open

WANDY666 wants to merge 3 commits into
mainfrom
openai-chat-output-logprobs

Conversation

@WANDY666

@WANDY666 WANDY666 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

改动

  • /v1/chat/completions 支持 logprobs 和 top_logprobs,流式和非流式都能返回生成 token 的概率。开启 logprobs 后,可以选择返回每个位置的 0–20 个候选。
  • 混合 batch 里,Triton 按请求直接读取原 logits,省掉整行词表复制。随机采样的 K=0 路径只统计排名。候选按各请求的实际 K 打包,再批量异步拷到 CPU。
  • MTP 根据接受掩码对齐候选。PD 首 token 从实际采样的节点取候选,覆盖 D 端完整命中以及 P 端完成 prefill 两种路径。

验证

  • 合入 main 的 5e9a7e66a 后,在 H200 上重跑 API、GPU 采样、结构化输出、共享内存和 PD 回归,294 项全部通过。
  • Black、Flake8 和 git diff --check 通过。

相关讨论 #1124。

  Add logprobs and top_logprobs to streaming and non-streaming responses.
  Use Triton to process requested logits rows directly and skip top-k
  selection when only the sampled token score is requested.
  Pack candidates by request and preserve alignment through MTP filtering.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant