singhpk234 commented on PR #10292: URL: https://github.com/apache/iceberg/pull/10292#issuecomment-2108568399
> According to this PR https://github.com/apache/iceberg/pull/7914, it doesn't seem that delete_orphan_files supports S3FileIO. If/when it does, it still might not work with S3 Express due to it requiring a / at the end of a ListObjects request, whereas S3 Standard does not. We could change s3FileIO.listPrefix to add the / if it's not already present, but that could potentially break usages where a list request is required without a trailing delimiter. (E.g when the list request is for a partial prefix) If we claim, S3 express is supported with iceberg we have to make sure it works with all of the iceberg unless / until we can have a dedicated page to document limitation, my questions are mostly comming from there as i see the integ test being updated but i am not able to figure what does fixing integ test imply, are we claiming S3 express is supported with iceberg ? > With regards to HadoopFileIO, does that refer to this? https://iceberg.apache.org/docs/1.5.0/aws/#hadoop-s3a-filesystem yes, HadoopFileIO can use any of the FS impl available, the link you shared is when using S3A, we also have EMR-FS using the HadoopFile system. HadoopFileIO default with i think HiveCatalog. > Is this still recommended for use? It depends, from what i am aware this is not prod recommended, since S3FileIO is the one we would be using. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
