Skip to content

Fix response byte accounting after partial TCP writes - #509

Merged
mathieucarbou merged 5 commits into
ESP32Async:mainfrom
labodj:codex/fix-partial-response-accounting
Nov 11, 2026
Merged

mathieucarbou merged 5 commits into
ESP32Async:mainfrom
labodj:codex/fix-partial-response-accounting

Conversation

@labodj

@labodj labodj commented Nov 10, 2026

Copy link
Copy Markdown

When AsyncClient::add() accepts only part of a pending response buffer, AsyncAbstractResponse::write_send_buffs() advances the buffer offset and breaks before adding the accepted bytes to payloadlen. For example, accepting 7 of 20 bytes queues those 7 bytes but returns 0 and leaves _writtenLength and the in-flight accounting behind.

Move payloadlen += added_len immediately after add(), so every accepted byte is counted before the partial-write branch. Buffer retention and subsequent retries continue to use the existing code.

The regression test compiles the actual production method with a controllable TCP peer. It checks zero acceptance and acceptance limits of 1, 7, 19 and 20 bytes, with chunk in-flight tracking both enabled and disabled. It verifies the returned byte count, cumulative written bytes, in-flight bytes, retained buffer offset and exact final payload. Eight of the ten scenarios fail against unchanged main; all ten pass with this fix.

Native tests pass with both GCC and Clang under AddressSanitizer and UBSan. Run them with:

python3 -m unittest discover -s tests -p 'test_response_*.py' -v
CXX=clang++ python3 -m unittest discover -s tests -p 'test_response_*.py' -v

The included workflow runs the same tests with both compilers. The TCP double exercises the short-write contract; it does not emulate lwIP or the radio.

Build verification

Local builds passed for:

Target Board Example
ESP8266 / Arduino 3.1.2 d1_mini PerfTests
ESP32 / Arduino 2.0.17 esp32dev PerfTests
ESP32 / Arduino 3.3.11 esp32dev PerfTests
ESP32-C3 (RISC-V) / Arduino 3.3.11 esp32-c3-devkitm-1 PerfTests
RP2040 / Arduino-Pico 4.4.4 rpipicow PerfTests
RP2350 / Arduino-Pico 4.4.4 rpipico2w PerfTests
BK7231N / LibreTiny 1.13.0 generic-bk7231n-qfn32-tuya AsyncResponseStream
RTL8710BN / LibreTiny 1.13.0 generic-rtl8710bn-2mb-788k AsyncResponseStream

ESP32 Arduino 3.3.11 was built from the official git tag with the matching IDF b774170f libraries. LibreTiny was built in an isolated Python 3.12 / PlatformIO 6.1.19 environment. These are compilation checks; no boards were flashed.

@mathieucarbou mathieucarbou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@labodj : we won't accept an AI-generated PR fixing a maybe issue (because it does not take into account how lwip / async_tcp works behind).
Please get rid of all this AI crap, and work on a real reproductible test case like we have in our examples that can prove the issue is real and reproductible with an ESP32.

@mathieucarbou

Copy link
Copy Markdown
Member

@willmmiles : I will let you verify - this relates to how lwip works so you'll be better at answering this PR ;-) Thanks!

@labodj

labodj commented Nov 11, 2026

Copy link
Copy Markdown
Author

Thanks for the review.
I tested this on an ESP32 with the existing LargeResponse example, instrumenting AsyncClient::add() results.
Across runs with up to 1,000 requests, 32 concurrent clients and 5% packet loss, I saw calls that accepted zero bytes, but none that accepted a positive partial length.
I cannot provide a hardware reproducer for the condition this PR fixes, so I’m closing it.
Sorry for submitting the proposed fix without that evidence.

@labodj labodj closed this Nov 11, 2026
@mathieucarbou

Copy link
Copy Markdown
Member

@labodj fyi @willmmiles did some verifications and the issue is valid also on esp32 - we will fix it through another pr. Thanks for letting us know about it!

@mathieucarbou mathieucarbou added Type: Bug Something isn't working and removed Status: Awaiting triage labels Nov 11, 2026
@mathieucarbou mathieucarbou reopened this Nov 11, 2026
@mathieucarbou

mathieucarbou commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

I tested this on an ESP32 with the existing LargeResponse example, instrumenting AsyncClient::add() results.
Across runs with up to 1,000 requests, 32 concurrent clients and 5% packet loss, I saw calls that accepted zero bytes, but none that accepted a positive partial length.

Right this one is harder to test and depends on lwip behaviour.

But the fix is still valid: each time we call add or write we should increase the payloadlen and it impacts _writtenLength.

But on ESP32, _writtenLength is not used => will add a commit to the PR to clean that.

And payloadlen only impacts the _in_flight count.

@mathieucarbou
mathieucarbou marked this pull request as ready for review November 11, 2026 21:34
@mathieucarbou mathieucarbou added Type: Improvement Type: Bug Something isn't working and removed Type: Bug Something isn't working Type: Improvement labels Nov 11, 2026
@mathieucarbou
mathieucarbou merged commit 3c1047a into ESP32Async:main Nov 11, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status: Pending Merge Type: Bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants