andygrove opened a new pull request, #7110:
URL: https://github.com/apache/datafusion-comet/pull/7110
## Which issue does this PR close?
N/A. This is build hygiene and is not tied to an issue.
## Rationale for this change
A clean `./mvnw clean test-compile` on the default profile (Spark 4.1, Scala
2.13) reports about 50 compiler warnings in the `spark` module. They bury new
warnings and hide real problems, so this clears the ones that can be cleared
without breaking the Spark 3.4/3.5 (Scala 2.12) builds.
## What changes are included in this PR?
- `SparkThrowable.getErrorClass` (deprecated since Spark 4.0): tests read
the error condition through a new `ShimSparkErrorCondition.errorCondition` in
`spark-3.x` / `spark-4.x`, because Spark 3.x has no `getCondition`.
- `scala.collection.JavaConverters` replaced with
`scala.jdk.CollectionConverters`. `scala-collection-compat` is already a
dependency, so this also compiles on 2.12.
- `mutable.Map.retain` in `CometCelebornShuffleManager` replaced by `--=`
over the matching keys. `filterInPlace` does not exist on 2.12.
- Arrays passed to Scala varargs parameters (`CometCelebornShuffleReader`,
`CometCelebornShuffleReaderSuite`) now go through `Array`-typed helpers instead
of copying with `: _*`.
- `Predef.any2stringadd` (`dir + "/data"`) replaced with string
interpolation, and the missing `()` added in `CometExistenceJoinBenchmark` and
`CometCelebornShuffleReaderSuite`.
- `CometTPCDSQuerySuite` no longer uses an early initializer.
`CometTPCDSQueryTestSuite` exposes `registerTpcdsQueryTests()`, which the
subclass calls at the end of its body, after its `tpcdsQueries` overrides are
initialized.
- `CometVector` reaches `sun.misc.Unsafe` reflectively. javac warns on every
compile-time reference to it and `@SuppressWarnings("sunapi")` is not honored.
- `DetectionDelegatingFileIO.serializeConfWith` is marked `@deprecated`,
which silences the warning for its call to the deprecated Iceberg delegate.
`RESTCatalogAdapter` gets `@SuppressWarnings` for Iceberg APIs whose
replacements are missing in older supported Iceberg versions.
Not fixed, and why:
- `Stream` / `toStream` in `AliasAwareOutputExpression` (4 warnings):
`LazyList` does not exist on Scala 2.12, and `@nowarn` would itself warn there
as unused under `-Pstrict-warnings`.
- javac "Cannot find annotation method 'value()' in type 'KeyFor'" from
`arrow-memory-core`: the optional `checker-qual` annotations are not on the
classpath.
- Maven model warnings (`artifactId` contains an expression, missing
`exec-maven-plugin` version, absent `src/main/scala` directories): these come
from the POM structure, not the code.
## How are these changes tested?
- `./mvnw clean test-compile` on the default profile and on `-Pspark-3.5`
(Scala 2.12): both succeed, and the only remaining warnings are the ones listed
above.
- Ran `SparkErrorConverterSuite`, `CometUnifiedShuffleMemoryAllocatorSuite`,
`CometPlanEqualitySuite` and `CometCelebornShuffleReaderSuite` on Spark 4.1.
The first three pass. In `CometCelebornShuffleReaderSuite` 21 tests fail with
`UnsatisfiedLinkError` or "native not loaded" because my worktree has no native
library, so CI needs to run it.
- Ran `CometTPCDSQuerySuite` against empty dummy table directories to check
the refactor: it registers 134 tests (99 v1.4, 34 v2.7, 1 extended). The real
queries need `SPARK_TPCDS_DATA`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
