GitHub user Xuanwo added a comment to the discussion: Ideas: add directory support for `FileIO`
Hi, thank you @dentiny for starting this. The topic of a file-based catalog has come up multiple times throughout Iceberg's history and has been rejected for similar reasons: - Iceberg is designed to avoid performing list object operations on storage services. It doesn't depend on storage services to provide listing support, which is one of the reasons it outperforms the Hive table format. - The filesystem catalog itself offers only very limited features. For example, users cannot rename a table atomicly. The latest discussion on this topic can be found at https://lists.apache.org/thread/v7x65kxrrozwlvsgstobm7685541lf5w. Some people are exploring this direction (although not in exactly the same way) at https://olympiaformat.org. You might find it interesting. --- Let’s return to our current discussion. I personally agree that there are some interesting use cases for an FS-based catalog. For example, if the layout on S3 is organized in the way you mentioned, users can read from this catalog directly without needing to connect to other services. (Note: This is an implementation detail regarding how other catalogs organize namespaces and tables. It’s possible for them to store the entire namespace in a Postgres database.) I think the way forward for us is as follows: We can expose a `list` operation in `FileIO`. This isn't being added solely for this discussion; our compaction and vacuum features will require this functionality as well. You could then implement an FS catalog outside of iceberg-rust as a third-party catalog. What do you think? GitHub link: https://github.com/apache/iceberg-rust/discussions/1246#discussioncomment-12958750 ---- This is an automatically sent email for issues@iceberg.apache.org. To unsubscribe, please send an email to: issues-unsubscr...@iceberg.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
