Skip to content

feat: add common transport-level handshake for RDMA and UBSHM - #47

Open
zchuango wants to merge 38 commits into
masterfrom
transport-handshake
Open

zchuango wants to merge 38 commits into
masterfrom
transport-handshake

Conversation

@zchuango

Copy link
Copy Markdown

No description provided.

Spicy-cream and others added 28 commits October 15, 2026 19:32
…lity

Resolve UBSHM handshake conflicts using the tested rebased tree. Retain backend-specific format negotiation, remote shared-memory lengths, and upstream IPC V2 tests.
Resolve current master conflicts and preserve UBSHM IPC V2 negotiation
Resolve master conflict and preserve shared memory name safety
…refresh

Resolve modular CMake conflict and preserve handshake protobuf generation
* Add FlatBuffers message and service support

Add IOBuf-backed messages, allocator-aware builders, and stable service
descriptors. Preserve shared-buffer copy semantics and validate allocator,
parsing, descriptor, and ownership contracts with the core unit suite.

Keep FlatBuffers 25.2.10 support disabled by default across CMake, Make,
and Bazel. Isolate generated test headers and select matching GoogleTest
headers and libraries. Network transport and Channel/Server integration
remain in the follow-up PR.

* Add FlatBuffers service generator and acceptance tests

Generate service and stub bindings with the upstream FlatBuffers parser.
Match flatc type escaping, preserve schema service identities, and reject
collisions with schema declarations, generated API names, and reserved
C++ identifiers. Publish header/source pairs with locking and rollback.

Cover imported and nested schemas, compiler modes, relocation, concurrent
generation, failure recovery, and real runtime dispatch. Pass compiler
flags through acceptance checks and pin validation to flatc 25.2.10.

* Verify FlatBuffers support in Linux and macOS CI

Run the core unit suite, codegen acceptance, and runtime tests through a
shared verification entry. Pin FlatBuffers and GoogleTest sources and
reject empty or skipped core test runs.

Exercise GCC 11 and Clang 14 on Linux and Protobuf 21 and 29 on macOS.
Build checksum-pinned Gperftools in the Linux runner temporary directory
without libunwind to avoid conflicts with hosted runner packages.

* Document FlatBuffers support

Add Chinese and English guides for messages, builders, descriptors, and
service generation. Describe shared-buffer ownership, verification,
allocation failures, supported dependency versions, and build options.

Document generator naming rules, failure recovery, acceptance tests, and
the shared CI verification entry. Keep network transport and
Channel/Server integration explicitly outside this PR.

* Document using other FlatBuffers versions

Describe how to move off the default 25.2.10: build the target version
into a private prefix, update the runtime, generator, and codegen test
version checks, and rerun verify.sh against that prefix. Use 23.5.26 as
the worked example and limit it to CMake builds.

List FlatBuffers among the supported dependencies in the getting started
guides, following the format used for protobuf and libunwind.

---------

Co-authored-by: xulei25 <xulei25@baidu.com>
…#3637)

* Fix macOS futex wake accounting with per-waiter notifications

* Fix ut

* Guard macOS futex waiter cleanup and fix timeout test diagnostic
* Fix JSON parsing offsets at EOF

* Opt

* Fix ut
* Fix ResourcePool block item count race

* Optimize ResourcePool nitem publication
* fix(bthread): synchronize keytable pool access

Read pool pointers and the destroyed state under the existing rwlock. Keep only a boolean availability hint between the read and write lock sections, then recheck the pool state under the write lock.

Add coverage for concurrent borrowing, returning, and pool destruction, including reclamation of tables returned after destruction.

* test(bthread): ensure keytable return after pool destruction

* test(bthread): isolate keytable pool test flag
Retain the in-flight I/O delay and release the transaction synchronously when deferred cleanup cannot be scheduled. Unlink before handing cleanup ownership away and detach the closed ring from the transaction pool slot.

Cover timerfd creation failure in active and callback cleanup paths, including shared-memory release and repeated close. Validated 177 handshake/RDMA/UBSHM tests and 83 ASan tests.
Keep transaction slots until both the connection and cleanup callbacks release their references. Pin device-event lookups under the manager lock and route passive and timed-out closes through one cleanup owner.

Preserve UBS mapping metadata during unlink, reject non-positive cleanup delays, and complete the in-flight I/O wait after signal interruptions.

Validate with 182 handshake/RDMA/UBSHM tests, 88 ASan tests, and 10 repeated rounds of the three lifetime regression cases.
Store the owning SocketId in queued poll events and acquire its reference before accessing the endpoint. Ignore events for an old poller after the endpoint resets.

Reproduced the previous owner-release interleaving with ASan. Validate the fixed path with 54 UBSHM ASan tests, 54 regular UBSHM tests and 37 common handshake tests.
Transfer failed mapping cleanup to tasks that retain their remaining unmap and deallocate operations. Release transaction references after handoff, clean both sides independently, and stop retry callbacks before SDK finalization.

Add eight SDK fault-injection regressions covering slot reuse, independent retries, duplicate requests, rejected handoff, and shutdown ordering. Validation: 191 regular tests, 97 ASan tests, and 80 repeated ASan regression executions passed.
Serialize named allocation with cleanup retries and keep existing SDK objects intact. Reserve rollback storage before allocation, retain transaction ownership before mapping resources, and route allocation failures through the abort cleanup path.

Replace the retry counter race with a callback barrier and synchronized transaction inspection. Add name reuse, failed mapping, partial allocation, and rejected rollback regressions. Validation: 196 regular tests, 102 ASan tests, and 260 repeated ASan regression executions passed.
…ship

Register shared-memory names before SDK calls and defer device-event cleanup until queues are initialized. Abort construction after a recorded fault and publish ready resources under the transaction-manager lock.

Track name release independently from mapping cleanup. Serialize unlink with resource handoff, unmap already unlinked resources without deleting reused names, and leave peer names to their creators. Cover construction-time faults, active faults, unlink races, and UBS/IPC name reuse.

Validation: 203 regular tests, 109 ASan tests, and 400 repeated ASan regression executions passed with local SDK stubs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants