DengYiping opened a new pull request, #18392:
URL: https://github.com/apache/iceberg/pull/18392
A non-null VARIANT can contain a missing or null child, so zero parent nulls
and matching child bounds cannot prune `NOT_IN`, `NOT_EQ`, or `NOT_STARTS_WITH`
without null checks: these predicates match nulls under Iceberg's expression
API semantics.
Require non-null preservation before using parent null counts for
negative-predicate pruning, preserving primitive and supported transform
optimizations.
This fixes API pruning; it does not establish a Spark SQL wrong-result bug.
Follow-up: recognize `notNull(extract(...)) AND
negativePredicate(extract(...))` for the same path and type to recover safe
pruning, scoped to the conjunction and retaining NaN safeguards.
**Test plan:**
- RED:
- six regressions failed across metrics and content-stats evaluators;
equality controls passed.
- Verify a real Parquet writer produces equal child bounds despite a
missing child.
- GREEN:
- all 440 selected API/core/Parquet tests passed on JDK 17 and 21,
including primitive, transform, null, and NaN controls.
- Spotless, all six module Checkstyle tasks, and `git diff --check`
passed. Full-engine suites were not run.
---
**AI Disclosure**
- Model: gpt-6.1-sol
- Platform/Tool: Codex
- Human Oversight: human repro the problem locally. Fix is relatively small
(2 lines) and easy to review. Human forces TDD to be used in the development
- Prompt Summary: Fix VARIANT negative-predicate pruning with red-green TDD.
Production changes, regression tests, writer fixture
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
