This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.2-lts by this push:
new 10dcb4eb60 [fix](Outfile) fix bug that the fileSize is not correct
when outfile is completed (#23660)
10dcb4eb60 is described below
commit 10dcb4eb60e32d03eb668d35cf4687aafdbcfab1
Author: Tiewei Fang <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Wed Aug 30 15:06:57 2023 +0800
[fix](Outfile) fix bug that the fileSize is not correct when outfile is
completed (#23660)
From pr: #22951
1. fix that the fileSize is not correct when outfile parquet/orc file
format is completed.
2. max_file_size of orc file format must be multiple of 64MB:
The function add(RowBatch) of orc::Writer would not flush any pending data
to the output stream until the strip data is filled. And the defult of strip
size of orc file format is 64MB, so we can not get the correct _written_len
from orcOutputStream until 64MB data has been writen.
---
be/src/vec/runtime/vfile_result_writer.cpp | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/runtime/vfile_result_writer.cpp
b/be/src/vec/runtime/vfile_result_writer.cpp
index fe4994256a..93b60bc1bf 100644
--- a/be/src/vec/runtime/vfile_result_writer.cpp
+++ b/be/src/vec/runtime/vfile_result_writer.cpp
@@ -426,7 +426,11 @@ Status
VFileResultWriter::_create_new_file_if_exceed_size() {
Status VFileResultWriter::_close_file_writer(bool done, bool only_close) {
if (_vfile_writer) {
_vfile_writer->close();
- COUNTER_UPDATE(_written_data_bytes, _current_written_bytes);
+ // we can not use _current_written_bytes to
COUNTER_UPDATE(_written_data_bytes, _current_written_bytes)
+ // because it will call `write()` function of orc/parquet function in
`_vfile_writer->close()`
+ // and the real written_len will increase
+ // and _current_written_bytes will less than
_vfile_writer->written_len()
+ COUNTER_UPDATE(_written_data_bytes, _vfile_writer->written_len());
_vfile_writer.reset(nullptr);
} else if (_file_writer_impl) {
_file_writer_impl->close();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
