Sigma-Ma commented on PR #174: URL: https://github.com/apache/hbase-connectors/pull/174#issuecomment-6872096842
> > While e2e testing these, I run into a situation where a query with IS NULL predicate still returned some rows where the given column wasn't null. > > Was checking this with Claude. the cause seems to be the due to StoreScanner returning mid-rows once hbase.storescanner.pread.max.bytes limit (256KB) is reached. This seems to cause the SparkSQLPushDownFilter.filterRow being called only on the second portion of the row, but when the cell we should check for null is in the first portion, the row gets in. > > I've checked disabling pread by setting hbase.storescanner.pread.max.bytes to -1, or also setting it to 1GB, and query works. > > Suggested fix by claude: > > 1. Override hasFilterRow to always return true in SparkSQLPushDownFilter (similar to hbase builtin filter SingleColumnValueFilter), so that RegionScannerImpl then moves the size and time limits to BETWEEN_ROWS; > 2. Only call setBatch when no pushdown filter is attached. HBasePartitionReader.buildScanner (line 180): > > ` if (batch > 0 && !hasNullCheck && pushDownFilter.isEmpty) scan.setBatch(batch)` Thanks, that was the missing bit. I’ve added hasFilterRow() to the shared filter and guarded setBatch in both Spark 3 and 4. I also added a regression with flushed wide rows and a low PREAD threshold, checking NULLs near both ends of the row. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
