Repository navigation
iperf3: Don't wait for a full block when receiving TCP data. - #1167
Conversation
9a505eb to
d58b242
Compare
|
@dpgeorge - ipref is much better now. |
|
Thanks for the PR. I've tested this on PYBD_SF2 and it does improve the situation with large TCP buffer size (tested with
Do you think this is a useful feature to add? It adds quite a bit of code and complexity, and could slow things down on some boards (eg when testing very fast gigabit Ethernet) because now there are loops over sockets rather than just one socket. Also, the feature doesn't work reliably for me. Sometimes it finishes but other times it times out at the end, eg on the host side: I suggest either drop the parallel-streams feature, or make it reliable (hopefully without any more complexity...). |
d58b242 to
417f2e5
Compare
|
Found the timeouts: after On complexity and speed: it no longer re-creates the listening socket (fixed backlog of 4), and finding the ready stream checks the first one directly, so single-stream Re-creating the listener also exposed a modlwip bug: closing a listening socket writes past its pcb and corrupted the control connection, which hung about 1 in 30 |
The TCP receive path read one whole block (the client's `-l` length) per poll event with a blocking `readinto()`, which on MicroPython returns only once the buffer is full. The sender is not required to end the stream on a block boundary, and the reference iperf3 client usually doesn't when its writes are larger than the socket can take at once: at the end of the test it abandons the block it is in the middle of writing and sends TEST_END. The server then waits forever for the rest of that block, never reads TEST_END from the control socket, and both ends hang. That makes `iperf3 -c <board>` with the default 128k block size stall on most runs, while `-l 500` works because such a small write is never split. Fix this by making the data socket non-blocking when it's used to receive TCP data, and counting however many bytes each read returns. The same applies to the client in reverse mode. Tested on an OPENMV_N6 over Ethernet with iperf 3.17.1 as the client and its default arguments: 2 of 12 runs completed before this change, 12 of 12 after. Also tested with iperf 3.9, 3.15, 3.16 and 3.22 as the client, with `-l 500` and `-R`, and with the board as the client (forward and reverse) against `iperf3 -s`. Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
The reference iperf3 client only sends "pacing_timer" in its parameters from version 3.2 onwards. With an older client the server raised `KeyError: pacing_timer` after the streams were created and the client hung. Use the same 1000ms default that this module's own client sends. Tested on an OPENMV_N6 over Ethernet with iperf 3.1.3 as the client. Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
The server accepted a single data connection whatever the client asked for, so `iperf3 -c <board> -P 2` never started: the client waited for its second stream to be accepted and the server waited for data on the first. Accept as many TCP data connections as the client's "parallel" parameter says, poll them all, and report the bytes transferred per stream in the results (the reference implementation numbers its streams 1, 3, 4, ...). The interval lines printed by the server remain the total of all streams. The client opens its streams back to back, so the listening socket now has a backlog of 4 rather than 1. That is also as many streams as there are TCP pcbs for with lwIP's default of 5, after the control connection. UDP and the client are unchanged, they still use one stream. Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
The reference iperf3 server closes its data streams as soon as the client sends TEST_END, before it asks for the client's results. This server kept them open until the end of the test. When the server is receiving, the data the client still had in flight stays queued in the network stack, because the server no longer reads it. With two streams that can be all of the stack's receive buffers (on OPENMV_N6 two streams can hold 48 segments, against a pool of 32), and then the client's results, or its acknowledgement of the server's results, can't be received. `iperf3 -c <board> -l 5000 -P 2` then timed out at the end: on OPENMV_N6 in 10 of 10 runs over WiFi and 1 of 10 over Ethernet. When the server is sending, a client that reads whole blocks with a blocking read (iperf 3.1.3) is left waiting for the rest of a block and doesn't see the request for results. The server used to keep sending until its socket buffer was full, which is not enough once the client has stopped reading. With the streams closed, the client sees the end of the stream instead. Close the streams on TEST_END, as the reference implementation does. Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
dpgeorge
left a comment
dpgeorge
left a comment
There was a problem hiding this comment.
Thanks for updating. I tested on OPENMV_N6 and parallel streams works reliably now.
During merge I'll bump the version to 0.2.0. A minor bump is warranted now that parallel streams are supported.
Previous commits fixed bugs and implemented parallel stream support. Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
417f2e5 to
00f72c4
Compare
Summary
Fixes to the
iperf3server so it works with the reference iperf3 client's default arguments, with parallel streams, and with old clients. Found by running iperf 3.1.3 to 3.22 against an OPENMV_N6.Don't wait for a full block when receiving TCP data.
iperf3 -c <board>with default arguments (128k blocks) hangs on most runs. The TCP receive path reads one whole block per poll event with a blockingreadinto(), which on MicroPython returns only once the buffer is full. The reference client usually ends the stream part-way through a block and then sendsTEST_END, so the server waits forever for the rest of the block and never readsTEST_END. The data socket is now non-blocking when receiving TCP data, and each read counts however many bytes it returns. The same applies to the client in reverse mode.Accept clients that do not send a pacing timer. The reference client only sends
pacing_timerfrom 3.2 onwards; with an older client the server raisedKeyError. It now defaults to 1000 ms.Support parallel TCP streams in the server. The server accepted one data connection whatever the client asked for, so
-P 2never started. It now accepts as many as the client'sparallelparameter says, polls them all, and reports bytes per stream in the results (the reference implementation numbers its streams 1, 3, 4, ...). The listening socket has a backlog of 4 so the client's streams can queue; that's also as many streams as lwIP's default of 5 TCP pcbs leaves room for.Close the data streams when the test ends. The reference server closes its streams on
TEST_END, before asking for results. This one kept them open, so with parallel streams their unread data could fill lwIP's receive buffers and the results exchange couldn't get through. Closing them also lets a client that reads whole blocks (iperf 3.1.3) in reverse mode see the end of the stream, which replaces the server's old "keep writing" loop.Testing
OPENMV_N6 on MicroPython master, Ethernet and WiFi, iperf3 client on a Linux host.
-l 5000 -P 2, before this revision-l 5000 -P 2-t 3-R,-l 500-P 2 -R,-P 4,-P 4 -l 500-b 20M, both directions-R,-P 2,-P 2 -R,-l 500-RSingle-stream throughput with
-l 500over Ethernet, where per-read overhead shows: 35.5 Mbit/s before this PR, 35.8 after (medians of 6).Not addressed:
-P 5needs 6 TCP connections and the port allows 5.Trade-offs and Alternatives
Reads are now as large as whatever has arrived rather than one full block, so there are more Python-level reads per second with large blocks; throughput did not drop.
UDP and the client side still use a single stream.