This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ee2eeb70a58 [SPARK-45283][CORE][TESTS] Make StatusTrackerSuite less
fragile
ee2eeb70a58 is described below
commit ee2eeb70a58990eb1fe25a47babff0d87319389d
Author: Bo Xiong <[email protected]>
AuthorDate: Tue Oct 3 19:01:46 2023 -0500
[SPARK-45283][CORE][TESTS] Make StatusTrackerSuite less fragile
### Why are the changes needed?
It's discovered from [Github
Actions](https://github.com/xiongbo-sjtu/spark/actions/runs/6270601155/job/17028788767)
that StatusTrackerSuite can run into random failures, as shown by the
following error message. The proposed fix is to update the unit test to remove
the nondeterministic behavior.
```
[info] StatusTrackerSuite:
[info] - basic status API usage (99 milliseconds)
[info] - getJobIdsForGroup() (56 milliseconds)
[info] - getJobIdsForGroup() with takeAsync() (48 milliseconds)
[info] - getJobIdsForGroup() with takeAsync() across multiple
partitions (58 milliseconds)
[info] - getJobIdsForTag() *** FAILED *** (10 seconds, 77 milliseconds)
[info] The code passed to eventually never returned normally.
Attempted 651 times over 10.005059944000001 seconds.
Last failure message: Set(3, 2, 1) was not equal to Set(1, 2).
(StatusTrackerSuite.scala:148)
```
Full trace can be found
[here](https://issues.apache.org/jira/browse/SPARK-45283).
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```
build/mvn package -DskipTests -pl core
build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.StatusTrackerSuite
test
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43194 from xiongbo-sjtu/master.
Authored-by: Bo Xiong <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
b/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
index 0817abbc6a3..9019ea484b3 100644
--- a/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/StatusTrackerSuite.scala
@@ -140,16 +140,19 @@ class StatusTrackerSuite extends SparkFunSuite with
Matchers with LocalSparkCont
}
sc.removeJobTag("tag1")
+
// takeAsync() across multiple partitions
val thirdJobFuture = sc.parallelize(1 to 1000, 2).takeAsync(999)
- val thirdJobId = eventually(timeout(10.seconds)) {
- thirdJobFuture.jobIds.head
+ val thirdJobIds = eventually(timeout(10.seconds)) {
+ // Wait for the two jobs triggered by takeAsync
+ thirdJobFuture.jobIds.size should be(2)
+ thirdJobFuture.jobIds
}
eventually(timeout(10.seconds)) {
sc.statusTracker.getJobIdsForTag("tag1").toSet should be (
Set(firstJobId, secondJobId))
sc.statusTracker.getJobIdsForTag("tag2").toSet should be (
- Set(secondJobId, thirdJobId))
+ Set(secondJobId) ++ thirdJobIds)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
