lymerin opened a new pull request, #7421:
URL: https://github.com/apache/shenyu/pull/7421
Fixes #7417
## Problem
`DividePluginCases.testRocketMQHello` starts a push consumer and then sends
the first request that can create `shenyu-access-logging`. `consumer.start()`
does not mean the consumer has acquired that topic’s queues. A rebalance round
can learn the new route after reading an empty cached queue set; with
RocketMQ’s default 20-second rebalance interval, the next assignment can occur
after the test’s 30-second consumption deadline.
We reproduced this on the unchanged pre-fix commit
`a12aa3e595e59f2b82ea2e73835770fe1fdcb88a`, which already includes the
rule-sync and collector-startup fixes. With a controlled 1200 ms delay on the
HTTP example’s outgoing response, the original test timed out in HTTP,
WebSocket, and ZooKeeper sync modes (3/3). Each request returned HTTP 200, and
an independent broker read recovered its exact access log. In the traced
WebSocket run, the broker stored the message at consumer-start +1.36 s; the
consumer learned the route at +20.77 s but still had no target queue when the
assertion expired.
Issue #7417 contains the reproduction steps, log excerpts, and related CI
jobs. Those historical jobs show the same timeout symptom; their logs do not
establish that every failure had the same broker and consumer state as the
controlled reproduction.
## Changes
1. **Wait for consumer readiness before sending the request.**
`RocketMQTestSupport` starts an admin client, creates the access-log topic if
absent, checks for a readable and writable route, starts the test consumer, and
waits until it owns every expected target-topic queue. A retry-topic queue
cannot satisfy this check. Preparation shares one 60-second deadline. The
existing 30-second consumption timeout and RocketMQ’s default rebalance
settings are unchanged.
2. **Match the log from this run.** Each run uses a unique consumer group
and request ID. The listener parses the access-log JSON and requires the exact
request path and query plus HTTP status 200. An old or unrelated message cannot
make the test pass.
3. **Report where a timeout occurred.** Before the request, the helper
records broker queue offsets. On a consumption timeout, it reports the last
preparation state, current consumer queues, and broker offsets, then
independently reads newly stored messages. A failure in one diagnostic query
does not prevent the others. Exception summaries remain on one line for CI
searches, with the original causes retained.
4. **Preserve accurate failures.** HTTP request exceptions are labeled as
HTTP failures. Resource cleanup preserves the primary test failure if consumer
shutdown also fails; a shutdown failure after an otherwise successful case
still fails the test. Focused unit tests cover queue readiness and exact log
matching.
## Dependency and scope
`rocketmq-tools:4.9.3` is test-scoped. Its public `DefaultMQAdminExt` APIs
prepare the topic, inspect consumer queue assignments, and inspect broker
offsets; `rocketmq-client` does not provide those admin operations.
`logback-classic` is excluded to retain the existing logging binding.
Transitive `commons-codec` is excluded so the existing E2E HTTP dependency
continues to resolve version 1.11 instead of 1.9. `fastjson:1.2.76` was already
present through `rocketmq-client`; this PR does not add or upgrade it.
Explicit topic preparation changes this case’s coverage: it tests access-log
collection, broker storage, and consumption **after topic preparation**. It no
longer exercises first-message automatic topic creation. The helper creates
four queues if the topic is absent and reuses a valid existing queue count. The
readiness check assumes this E2E case’s single-broker Compose topology.
Multi-broker support and the Kafka sibling test’s matching behavior are
separate work.
## Verification
Development runs used JDK 17, the repository Maven wrapper with the
independent `shenyu-e2e/pom.xml` reactor, RocketMQ broker 4.4.0, and client
4.9.3.
| Check | Result and limit |
| --- | --- |
| Pre-fix source, controlled 1200 ms backend delay | HTTP, WebSocket, and
ZooKeeper reached the original 30-second MQ assertion timeout (3/3). All three
exact HTTP 200 logs were independently read from the broker. |
| Readiness implementation, same controlled delay | All three modes passed
(3/3). Preparation took 1056/907/753 ms; measured upstream response times were
1221/1217/1214 ms. These runs preceded the final failure-label and cleanup
refinements. |
| Ordinary HTTP E2E after failure-path refinements | Eight module tests
passed; the exact request log was stored and consumed. Preparation took 1142
ms. The final diagnostic-formatting edit came afterward. |
| Final-source module checks | Four `RocketMQTestSupportTest` tests and
Checkstyle passed. A real RocketMQ exception containing a newline produced a
one-line diagnostic while retaining its original cause. |
| Dependency and license checks | RAT passed. Dependency inspection retained
`commons-codec:1.11`, found no added Logback binding, and left fastjson at its
pre-existing version. Later edits did not change the POM or license headers. |
| Failure diagnostics | An offline-consumer probe still reported broker
offsets and the stored message when consumer-state lookup failed. Missing-rule
and unreachable-MQ probes failed at their expected stages. HTTP and shutdown
failure paths were also exercised. These probes used copied classes outside the
repository and are not counted as gateway E2E runs. |
A later 1200 ms rerun measured only 793 ms of actual upstream time; a 2000
ms trial failed the HTTP precheck before reaching the MQ assertion. Neither is
counted in the delay-regression result.
The pushed commit `9f0d98a` was rebased onto the latest upstream `master`
after these runs and **has not been retested**. The full root build has not
been run. CI results and any further local runs should be recorded against this
commit.
Focused command, with the module’s Compose services running:
```bash
./mvnw -B -f shenyu-e2e/pom.xml \
-pl shenyu-e2e-case/shenyu-e2e-case-logging-rocketmq -am test \
-Dtest=DividePluginTest,LoggingRuleSyncTest,RocketMQTestSupportTest \
-Dsurefire.failIfNoSpecifiedTests=false
```
`shenyu-e2e` is an independent Maven reactor; the root build does not
execute these tests.
- [x] I have read the contribution guidelines.
- [x] I submitted tests covering the changed behavior.
- [x] My local test passed `./mvnw clean install -Dmaven.javadoc.skip=true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
