madeirak commented on issue #12761: URL: https://github.com/apache/iceberg/issues/12761#issuecomment-2795732698
> You can use `rewriteAll` to rewrite every data file. This will not skip already sorted files. Usually this is good, as based on the `max-file-group-size-bytes` multiple files are compacted in a group. The sorting happens on group level, and the resulting files will contain distinct ranges for the sorted columns. When a query uses filtering for the sorting columns most of the files could be filtered out on metadata level which could speed up the query massively. Thx for reply. I am sorting the data indeed to accelerate query execution. Currently, the partition of this Iceberg table is date. I want to have a way to make almost all the data in order. This can speed up the query speed of large table (PB level). At present, the problem encountered when `using rewrite_data_files procedure` is that `min-file-size-bytes` and `max-file-size-bytes` need to be set. When some newly written files are between `min-file-size-bytes` and `max-file-size-bytes`, they will not be sorted. But at the same time, because it is a large table, PB level, I don’t want to run the full sort frequently every time. Is there a better way? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
