liuliquan-marshal opened a new pull request, #16928: URL: https://github.com/apache/iceberg/pull/16928
see step 3 in #16863 # code change summary 1. Use concurrent multipart uploads rather than whole-file serial putObject. `OSSOutputStream.java` 2. Add relative properties. `OSSOutputStream.java` 3. Add unit tests for `OSSOutputStream`. Code coverage of `OSSOutputStream` is 92% now. # performance compare ## BEFORE ``` Benchmark (bufferSizeKB) (fileIOClass) (fileSizeKB) Mode Cnt Score Error Units FileIOBenchmark.sequentialWrite 1024 org.apache.iceberg.aliyun.oss.OSSFileIO 1048576 avgt 5 4162.820 ± 162.809 ms/op FileIOBenchmark.sequentialWrite 1024 org.apache.iceberg.aws.s3.S3FileIO 1048576 avgt 4 1615.085 ± 73.897 ms/op ``` ## AFTER ``` Benchmark (bufferSizeKB) (fileIOClass) (fileSizeKB) Mode Cnt Score Error Units FileIOBenchmark.sequentialWrite 1024 org.apache.iceberg.aliyun.oss.OSSFileIO 1048576 avgt 5 1103.280 ± 43.540 ms/op FileIOBenchmark.sequentialWrite 1024 org.apache.iceberg.aws.s3.S3FileIO 1048576 avgt 4 1615.085 ± 73.897 ms/op ``` OSSFileIO is about 31% faster than S3FileIO when sequentialWrite now. The reason is that 8MB default part size can provide higher concurrency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
