yangshangqing95 commented on issue #17632: URL: https://github.com/apache/iceberg/issues/17632#issuecomment-5285154925
> I'm a little confused on how initial-default is related here. I think this is an issue whenever you add a new column? Hi @RussellSpitzer Yes, the underlying dictionary filter issue is not caused by the initial default itself. It can occur whenever a `FLOAT` or `DOUBLE` column exists in the current Iceberg schema but is absent from the Parquet file, and a `notNaN` predicate reaches `ParquetDictionaryRowGroupFilter`. The initial default is relevant to the end-to-end Spark reproduction: - Without an initial default, a newly added optional column has an implicit value of `null` for old files. Spark also pushes an `isNotNull` condition for `new_float <> NaN`, so the old file can be eliminated by the metrics filter before dictionary filtering. - With a non-NaN initial default, old rows logically have that default value. The metrics filter cannot eliminate the file, so the `notNaN` predicate reaches the dictionary filter and exposes the missing map entry. Therefore, adding a column alone does not necessarily produce a user visible failure. The lower level bug applies to any missing floating point column if `notNaN` reaches the dictionary filter; the initial default makes that path reachable in the Spark SQL reproduction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
