navinvishy commented on issue #12762: URL: https://github.com/apache/iceberg/issues/12762#issuecomment-3194542659
@slfan1989 we are also looking into this currently, and are facing issues similar to yours. Thanks for working on this. I had a related question: for tables with a very large number of partitions, we frequently move older partitions into infrequent access storage classes on S3 such as deep archive. From what I can tell from the snapshot/migrate procedures, though they do not scan the entire files, they still require [reading footers](https://github.com/apache/iceberg/blob/main/data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java#L227) for metadata. This would be prohibitively expensive for files in deep archive. Is it possible to disable this for iceberg and collect no metadata? Could that be made an option in iceberg? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
