jbonofre commented on PR #9884: URL: https://github.com/apache/iceberg/pull/9884#issuecomment-1986747262
@danielcweeks I think it's not possible to Iceberg (or any project) to define/support all file schema on its own because: 1. From a legal standpoint, it's not possible (for instance, you want to support OVH Store that uses GPL licensed, which is Cat X dependency for ASF) 2. It's not possible to add and maintain all possible scheme file (like Wasabi, Levvia, etc) in Iceberg itself 3. We need extension point for let users override or adapt the file IO. It's similar to what I did in Beam IOs, in order to be flexible and extensible enough. I got your point about "well-defined/standardize", but it's up to the user who define his custom scheme. If he implements `DeleegateFileIO` correctly, it should work. The rationale behind this proposal is: 1. Increase Iceberg adoption by allowing user to define his own file scheme and file system providers to provide their "connector" 2. Make Iceberg a little more flexible about File IO by giving an option to the user to tweak/tuning existing file schemes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
