[
https://issues.apache.org/jira/browse/HBASE-30178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132969#comment-18132969
]
mazhengxuan commented on HBASE-30178:
-------------------------------------
While working on HBASE-30413, I noticed a few other possible follow-ups.
Perhaps we could track these separately:
# A query like {{key < 'm' OR key = 'a'}} can return {{a}} through both a Scan
and a Get. We could deduplicate points and remove those already covered by scan
ranges, with tests for both Spark 3 and Spark 4.
# Some filters are accepted for pushdown even though the encoder cannot handle
their types correctly, leading to wrong results or exceptions. We could leave
unsupported comparisons to Spark and avoid unsafe row-key pruning for those
types.
# Date conversion can shift the day in some time zones, and negative
timestamps round differently from Spark 3. We could align these conversions and
add round-trip tests while keeping the existing storage format.
# Selecting a missing column may drop a sparse HBase row instead of returning
it with null. This needs an end-to-end test for both Scan and Get, followed by
adjusting column pruning to preserve those rows.
# Spark 4 no longer falls back to SparkConf for {{{}cacheblocks{}}},
{{{}cachedrows{}}}, {{{}batchsize{}}}, and {{{}bulkget.size{}}}. We could
restore the Spark 3 precedence so existing application settings continue to
work.
# The JavaHBaseContext batch API is still missing from Spark 4, which leaves a
gap for Java users. We could port the wrappers and Java tests, reusing the
existing HBaseContext implementation.
# Changes to shared sources or parent POMs can miss Spark 4 CI because profile
activation currently depends on {{spark4}} appearing in the changed modules. We
could extend that check to cover shared dependencies too.
Does this split sound reasonable? I'd probably start with the first two, since
they affect query results.
> Add support to spark 4 in hbase-connectors
> ------------------------------------------
>
> Key: HBASE-30178
> URL: https://issues.apache.org/jira/browse/HBASE-30178
> Project: HBase
> Issue Type: Umbrella
> Components: hbase-connectors
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> With spark 4.0.0 GA for around a year now, we should start working on making
> hbase-connectors integrate with spark 4.
> Since we would still need to support spark 3 for a while and due to spark 4
> requiring minimum jdk17 and scala 2.13, it would be easier to define a
> specific spark 4 module under the hbase-connectors root, so that
> spark.version, scala.version, compiler flags, and optional spark-4 only APIs
> never collide with spark 3 in a single parent pom.
> We should keep feature parity as much as possible. Datasources, context, rdd
> and connection cache related logic can be duplicated under this new
> hbase-spark4 module, in order to compile and package with jdk17 and scala
> 2.13. This module build should be isolated on a specific maven profile, so it
> wouldn't be build by default.
> Pushdown filter related classes and filter protocol would remain on a common
> module (no code duplication needed).
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
