Fokko commented on issue #791: URL: https://github.com/apache/iceberg-python/issues/791#issuecomment-2156736837
For Arrow, the `binary` cannot store more than 2GB in a single buffer, not a single field. See [Arrow docs](https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) for more context. > This is interesting, why would Polars go with large_binary by default? See: https://github.com/pola-rs/polars/issues/7422 Still, I think the inconsistency is not good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
