zhangwl9 opened a new issue, #4186: URL: https://github.com/apache/amoro/issues/4186
### Search before asking - [x] I have searched in the [issues](https://github.com/apache/amoro/issues?q=is%3Aissue) and found no similar issues. ### What would you like to be improved? Amoro currently uses the IcebergRewriteExecutor to perform small-file merging. This executor is based on a “row-based rewrite” model: read Parquet files -> deserialize into Record objects -> reserialize -> write to new files. The core logic is located in the 'AbstractRewriteFilesExecutor#rewriterDataFiles()' method. Existing Issues: 1. CPU-intensive: A significant amount of CPU time is consumed by decompressing Parquet files, creating objects, and re-encoding and compressing them. 2. Memory pressure: The deserialization process generates a large number of temporary objects, increasing GC pressure. 3. Inefficiency: For scenarios where files are merely merged (without modifying data), row-based rewrite is a massive waste of resources. ### How should we improve? For Parquet-format files in the iceberg table, in scenarios where `deleteFile` is not called, this approach bypasses row-level read/write operations. By utilizing the Hadoop Parquet API’s `ParquetFileWriter.appendFile()` method to directly merge RowGroups, it skips all encoding and decoding steps, thereby achieving fast compaction of Parquet files. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Subtasks _No response_ ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
