TangSiyang2001 commented on code in PR #45036:
URL: https://github.com/apache/doris/pull/45036#discussion_r1872446463
##########
be/src/vec/exec/format/file_reader/new_plain_text_line_reader.cpp:
##########
@@ -325,13 +324,13 @@ Status NewPlainTextLineReader::read_line(const uint8_t**
ptr, size_t* size, bool
*eof = true;
return Status::OK();
}
- _line_reader_ctx->refresh();
int found_line_delimiter = 0;
size_t offset = 0;
bool stream_end = true;
while (!done()) {
// find line delimiter in current decompressed data
uint8_t* cur_ptr = _output_buf + _output_buf_pos;
+ _line_reader_ctx->refresh();
Review Comment:
This fix may result in bad case in very wide table, aka per line is larger
than read buffer size.
We previously not did so for the parse error case is low probability and for
performance reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
