[BUG]: Timing dependent concurrency flake via test_destroying_child_node_invalidates_embedded_handles #3156

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Summary

Two related CUDA graph lifetime tests failed intermittently on Windows free-threaded Python when pytest-run-parallel ran four copies of each test concurrently. Both failures disappeared when the jobs were rerun.

The evidence points to a possible raw-CUgraph handle reuse/ABA race in the process-wide graph registry. A child graph can be destroyed by CUDA before its registry entry is retired, allowing another thread to resolve a reused raw handle to stale metadata.

Failing run

Run: https://github.com/NVIDIA/cuda-python/actions/runs/39296344205
Commit: 9d903cdcaee82731618dd56e8c76bcd528918592

Failure 1

Job: https://github.com/NVIDIA/cuda-python/actions/runs/39296344205/job/118215572432

Environment:

• Windows
• Python 3.14.8 free-threaded
• CUDA toolkit 13.4.3
• A100 40 GB, MCDM
• Driver 596.36
• pytest-run-parallel==0.10.0
• --parallel-threads=4

Failing test:

tests/graph/test_graph_definition_lifetime.py::
test_destroying_child_node_invalidates_embedded_handles

The test failed during its first middle.embed(inner), before reaching its destruction or invalidation assertions:

GraphDefinition.embed
GraphNode.embed
GN_embed
HANDLE_RETURN(graph_clone_attachments(h_embedded, child_def._h_graph))
CUDAError: CUDA_ERROR_INVALID_VALUE

Source:

• Test:

  cuda-python/cuda_core/tests/graph/test_graph_definition_lifetime.py

  Lines 445 to 465 in 9d903cd

  	def test_destroying_child_node_invalidates_embedded_handles(init_cuda):
  	"""Destroying a child owner invalidates all embedded graph and node views."""
  	inner = GraphDefinition()
  	inner.callback(lambda: None)
  	middle = GraphDefinition()
  	middle.embed(inner)
  	outer = GraphDefinition()
  	outer_child = outer.embed(middle)
  	embedded_middle = outer_child.child_graph
  	embedded_child = next(node for node in embedded_middle.nodes() if isinstance(node, ChildGraphNode))
  	embedded_inner = embedded_child.child_graph
  	embedded_callback = next(node for node in embedded_inner.nodes() if isinstance(node, HostCallbackNode))
  	outer_child.destroy()
  	assert not outer_child.is_valid
  	assert int(embedded_middle.handle) == 0
  	assert int(embedded_inner.handle) == 0
  	assert not embedded_child.is_valid
  	assert not embedded_callback.is_valid

• Embed implementation:

  cuda-python/cuda_core/cuda/core/graph/_graph_node.pyx

  Lines 1098 to 1117 in 9d903cd

  	with nogil:
  	HANDLE_RETURN(cydriver.cuGraphAddChildGraphNode(
  	&new_node, as_cu(h_graph), deps, num_deps, as_cu(child_def._h_graph)))
  	cdef cydriver.CUgraph embedded_graph = NULL
  	cdef GraphHandle h_embedded
  	try:
  	with nogil:
  	HANDLE_RETURN(cydriver.cuGraphChildGraphNodeGetGraph(
  	new_node, &embedded_graph))
  	h_embedded = create_child_graph_handle(
  	embedded_graph, h_graph, new_node)
  	HANDLE_RETURN(graph_clone_attachments(
  	h_embedded, child_def._h_graph))
  	except:
  	with nogil:
  	rollback_status = cydriver.cuGraphDestroyNode(new_node)
  	if rollback_status == cydriver.CUDA_SUCCESS:
  	invalidate_child_graph_state(h_graph, new_node)

• Attachment cloning:

  cuda-python/cuda_core/cuda/core/_cpp/rt/graph.cpp

  Lines 601 to 646 in 9d903cd

  	CUresult graph_clone_attachments(
  	const GraphHandle& h_clone,
  	const GraphHandle& h_source) {
  	if (!h_clone || !h_source) {
  	return CUDA_ERROR_INVALID_VALUE;
  	}
  	GraphBox* clone = get_box(h_clone);
  	GraphBox* source = get_box(h_source);
  	if (!clone->resource || !source->resource ||
  	!clone->attachments.empty()) {
  	return CUDA_ERROR_INVALID_VALUE;
  	}
  	// Build and rekey the clone metadata off-hierarchy so a CUDA mapping error
  	// cannot partially publish it.
  	GraphAttachmentMap attachments;
  	std::list<GraphBox> subgraphs;
  	StagedGraphMetadataList staged;
  	stage_graph_metadata(
  	*source, *clone, attachments, subgraphs, staged);
  	std::vector<GraphHandle> handles;
  	handles.reserve(subgraphs.size());
  	for (GraphBox& graph : subgraphs) {
  	handles.emplace_back(h_clone, &graph.resource);
  	}
  	CUresult status;
  	{
  	GILReleaseGuard gil;
  	status = rekey_graph_metadata(staged);
  	}
  	if (status != CUDA_SUCCESS) {
  	return status;
  	}
  	clone->attachments.swap(attachments);
  	if (subgraphs.empty()) {
  	return CUDA_SUCCESS;
  	}
  	clone->hierarchy->graphs.splice(
  	clone->hierarchy->graphs.end(), subgraphs);
  	graph_registry.register_handles(handles);
  	return CUDA_SUCCESS;

The identical matrix entry was rerun successfully:

https://github.com/NVIDIA/cuda-python/actions/runs/39296344205/job/118232342949

The rerun used the same source SHA, artifact digests, dependencies, GPU model, driver, and four-thread configuration.

Failure 2

A second free-threaded job failed a related test during the same run and attempt:

https://github.com/NVIDIA/cuda-python/actions/runs/39296344205/job/118215572511

Environment:

• Windows
• Python 3.15t
• CUDA 12.9.1
• V100, TCC
• --parallel-threads=4

Failing test:

test_destroying_conditional_node_invalidates_branch_handles[switch]

It failed before that invocation called owner.destroy():

RuntimeError: GraphDefinition is no longer valid

The traceback locals showed that two freshly returned branch objects were already invalid:

branches = [
    <GraphDefinition handle=0x0>,
    <GraphDefinition handle=0x0>,
    <GraphDefinition handle=0x212b886b810>,
    <GraphDefinition handle=0x212b886b3c0>,
]

Conditional branches use the same create_child_graph_handle() registry path as embedded child graphs:

Seeing newly created branch handles invalidated before their own owner is destroyed suggests cross-thread registry aliasing or invalidation.

Why this appears timing-dependent

ci/tools/run-tests enables four parallel threads on free-threaded Python:

pytest-run-parallel runs four concurrent copies of each parallel-safe test in the same process.

The failing and passing jobs used different pytest seeds, but run-tests passes --randomly-dont-reorganize, and these tests do not use randomness. The neighboring test order was identical.

The successful runner was also consistently slower, which likely changed the thread scheduling enough to avoid the race.

Suspected race

The likely interleaving is:

1. One test copy calls GraphNode.destroy().
2. GraphNode.destroy() calls cuGraphDestroyNode() first.
3. CUDA implicitly destroys the node's embedded child graphs.
4. The raw child CUgraph handle becomes eligible for reuse.
5. Before invalidate_child_graph_state() removes the old registry entry, another worker creates a child graph with the same raw handle value.
6. create_child_graph_handle() finds the old entry and returns its stale GraphHandle.
7. The first worker then retires/nulls that handle, or the second worker passes mismatched metadata to graph_clone_attachments().
8. The second worker observes an invalid GraphDefinition or gets CUDA_ERROR_INVALID_VALUE.

Relevant ordering in GraphNode.destroy():

Registry lookup and registration:

Registry retirement:

The registry has a mutex, but it protects individual lookup/register/unregister operations. It does not make the driver destruction → registry retirement sequence atomic.

The graph attachment design document explicitly notes that registry removal is needed to prevent corruption when CUDA reuses a raw handle:

Each parallel test invocation creates distinct graph objects, so this does not appear to be unsupported concurrent mutation of the same user-visible object.

Suggested investigation

Add temporary instrumentation around:

• create_child_graph_handle()
• retire_graph()
• invalidate_child_graph_state()
• graph_clone_attachments()

Log the following:

• Raw CUgraph value
• GraphBox address
• Parent hierarchy and owner node
• Thread ID
• Whether graph_clone_attachments() failed its local precondition or propagated an error from cuGraphNodeFindInClone()

A possible stress command on a matching Windows free-threaded runner is:

cd cuda_core
pytest -v --parallel-threads=4 --count=100 \
  tests/graph/test_graph_definition_lifetime.py::test_destroying_child_node_invalidates_embedded_handles \
  tests/graph/test_graph_definition_lifetime.py::test_destroying_conditional_node_invalidates_branch_handles

Possible fixes

The underlying fix likely needs one of:

• Serialize child-graph destruction/retirement against child-handle creation.
• Remove or reserve registry entries before CUDA can reuse the raw handle, with rollback if destruction fails.
• Make registry entries and removals generation-aware so an old owner cannot resolve or erase a newer graph using the same raw value.

As a short-term CI mitigation, the two invalidation tests could be marked thread_unsafe. However, that would hide what appears to be a real distinct-object concurrency defect rather than fix it.

How to Reproduce

Not reproducible. You could try the stress reproduction suggestions above.

Expected behavior

Under most circumstances, this test succeeds.

Operating System

No response

nvidia-smi output

No response