Sigma-Ma opened a new pull request, #177: URL: https://github.com/apache/hbase-connectors/pull/177
JIRA: https://issues.apache.org/jira/browse/HBASE-30513 ## What changes were proposed in this pull request? Restore datasource option > SparkConf > default precedence for these Spark 4 read settings: - `hbase.spark.query.cacheblocks` - `hbase.spark.query.cachedrows` - `hbase.spark.bulkget.size` Resolve these settings on the driver and pass the immutable values to partition readers through the serialized reader factory. Keep `hbase.spark.query.batchsize` behavior unchanged: per-read options still control cell batching, while the SparkConf value remains ignored. ## Why are the changes needed? Spark 4 currently reads these settings only from datasource options. Applications that configure them globally through SparkConf can silently use different read settings after upgrading from Spark 3. Restoring the existing precedence preserves global settings while allowing per-read overrides. ## How was this patch tested? ```bash mvn -B -ntp -Pspark4 -pl spark4/hbase-spark4 -am -DskipTests clean install mvn -B -ntp -Pspark4 -pl spark4/hbase-spark4 -Dsuites=org.apache.hadoop.hbase.spark.datasources.HBaseReadOptionsSuite,org.apache.hadoop.hbase.spark.datasources.HBaseReadOptionsReaderSuite,org.apache.hadoop.hbase.spark.datasources.HBaseScanBuilderSuite,org.apache.hadoop.hbase.spark.datasources.HBaseTableProviderSuite -Dsurefire.rerunFailingTestsCount=0 test ``` The new tests cover configuration precedence, defaults, invalid values, factory serialization, actual Scan parameters and Get batch sizes. They also verify that existing batchsize behavior is preserved for Scan and Get-only reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
