Skip to content

Fix the RocketMQ async-send callback NPE, and the client-java plugin with newer clients - #848

Open
wu-sheng wants to merge 2 commits into
mainfrom
fix/rocketmq5-nested-callback-npe
Open

wu-sheng wants to merge 2 commits into
mainfrom
fix/rocketmq5-nested-callback-npe

Conversation

@wu-sheng

@wu-sheng wu-sheng commented Nov 2, 2026

Copy link
Copy Markdown
Member

Fix the NullPointerExceptions of async send callbacks in the rocketMQ-5.x plugin (apache/skywalking#14189), and the rocketMQ-client-java-5.x plugin with newer clients

  • Add a unit test to verify that the fix works.
  • Explain briefly why the bug exists and how to fix it.

1. rocketMQ-5.x: two NPEs on every successful async send with a callback (apache/skywalking#14189)

Since rocketmq-client 5.1.4, all three async DefaultMQProducerImpl#send(..., SendCallback, ...) overloads wrap the callback in DefaultMQProducerImpl$BackpressureSendCallBack, whether backpressure is enabled or not. Only the wrapper reaches MQClientAPIImpl#sendMessage, so only the wrapper gets the SendCallBackEnhanceInfo. SendCallbackInstrumentation matches every SendCallback, so the application's callback is enhanced too, and the wrapper's onSuccess calls it on the same stack. Its OnSuccessInterceptor#beforeMethod throws an NPE before creating a span, its afterMethod stops the wrapper's span anyway, and the wrapper's afterMethod then throws an NPE on the empty context. OnExceptionInterceptor already handles callbacks without the info.

OnSuccessInterceptor now skips beforeMethod, afterMethod and handleMethodException when the callback has no info. The wrapper keeps the span and continues the context, and the inner callback runs inside that span, so each send gets one callback span, as it does with clients before 5.1.4. All three methods check the same field, and nothing sets it during the call, so they stay paired. Auto-batch (ProduceAccumulator) adds one more level of nesting and is handled the same way.

2. rocketMQ-client-java-5.x: push-consumer spans without peer since rocketmq-client-java 5.0.8

PushConsumerImplInstrumentation intercepted only the 7-argument PushConsumerImpl constructor, which the builder calls up to 5.0.7. The builder in 5.0.8 calls an 8-argument constructor, and 5.1.0+ a 9-argument one. So the MessageListener never got the endpoints, and consumer entry spans had no peer or mq.broker. The instrumentation now matches every constructor whose argument 0 is the ClientConfiguration and argument 3 is the MessageListener.

3. rocketMQ-client-java-5.x: lite topic and priority dropped from sent messages

To add its headers, the plugin rebuilt every message from the fields that 5.0 has, which dropped the lite topic (5.1.0+) and the priority (5.2.0+). Those messages went out as NORMAL messages, and sends to a PRIORITY topic failed with Current message type not match with topic accept message types. The send interceptors no longer change the message:

  • The producer copies every message into a PublishingMessageImpl on the sending thread, inside the exit span. A constructor interceptor injects the span into that copy's dynamic field.
  • MessageImpl#getProperties() returns a new map, which PublishingMessageImpl#toProtobuf puts into the user properties. An interceptor adds the headers to that map, for copies that carry them.

No reflection is involved, and the plugin no longer copies every message body.

Tests

  • rocketmq-scenario adds rocketmq-client 5.5.0. 5.5.1 has no broker image. All hook points are unchanged up to 5.5.1, and the plugin compiles against it.
  • rocketmq-5-grpc-scenario tested only rocketmq-client-java 5.0.5. It now tests 5.0.8, 5.1.0 and 5.2.2, the latest patch of each minor, on brokers 5.3.3, 5.3.4 and 5.4.0. The proxy of broker 5.5.0 rejects a gRPC consumer whose group doesn't exist yet, and this scenario relies on auto-created groups.

Locally (arm64), both scenarios pass with every new version, with the gRPC cases on brokers 5.3.2 and 5.3.4: the arm64 image of 5.4.0 lacks the RocksDB native library. With main's plugin, the 5.0.8 case fails, because the push-consumer spans have peer: '' and no mq.broker.

I also ran a throwaway end-to-end check with rocketmq-client-java 5.2.2 and broker 5.5.0, sending a message with priority 7 to a PRIORITY topic:

Agent Send Received
main fails, actualMessageType=NORMAL, acceptMessageTypes=[PRIORITY]
This PR succeeds priority=Optional[7], properties sw8, sw8-x
This PR, trace ignored succeeds priority=Optional[7], properties sw8-x

No unit tests or scenario code are added. Fix 1 isn't covered by a scenario: the scenarios send synchronously, and the reported trace looks the same with or without the NPEs.

…with newer clients

* rocketMQ-5.x: skip callbacks without SendCallBackEnhanceInfo in OnSuccessInterceptor.
  Since rocketmq-client 5.1.4 the producer wraps every async-send callback in
  BackpressureSendCallBack, and the wrapped callback threw an NPE and stopped the
  wrapper's span, whose afterMethod then threw a second NPE (apache/skywalking#14189).
* rocketMQ-client-java-5.x: intercept every PushConsumerImpl constructor taking the
  ClientConfiguration and the MessageListener, so push-consumer spans get their peer
  and mq.broker again with rocketmq-client-java 5.0.8 or later.
* rocketMQ-client-java-5.x: stop rebuilding the sent message, which dropped the lite
  topic and the priority. Inject the span into the producer's PublishingMessageImpl
  copy and add the headers to the properties it sends.
* Test rocketmq-client 5.5, and rocketmq-client-java 5.0, 5.1 and 5.2.
@wu-sheng wu-sheng added this to the 9.9.0 milestone Nov 2, 2026
@wu-sheng wu-sheng added the bug Something isn't working label Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant