elvestar commented on code in PR #30809:
URL: https://github.com/apache/doris/pull/30809#discussion_r1510645909
##########
be/src/vec/exec/format/csv/csv_reader.cpp:
##########
@@ -549,71 +549,12 @@ Status
CsvReader::get_parsed_schema(std::vector<std::string>* col_names,
}
Status CsvReader::_create_decompressor() {
- CompressType compress_type;
+ Decompressor* decompressor;
if (_file_compress_type != TFileCompressType::UNKNOWN) {
- switch (_file_compress_type) {
- case TFileCompressType::PLAIN:
- compress_type = CompressType::UNCOMPRESSED;
- break;
- case TFileCompressType::GZ:
- compress_type = CompressType::GZIP;
- break;
- case TFileCompressType::LZO:
- case TFileCompressType::LZOP:
- compress_type = CompressType::LZOP;
- break;
- case TFileCompressType::BZ2:
- compress_type = CompressType::BZIP2;
- break;
- case TFileCompressType::LZ4FRAME:
- compress_type = CompressType::LZ4FRAME;
- break;
- case TFileCompressType::LZ4BLOCK:
- compress_type = CompressType::LZ4BLOCK;
- break;
- case TFileCompressType::DEFLATE:
- compress_type = CompressType::DEFLATE;
- break;
- case TFileCompressType::SNAPPYBLOCK:
- compress_type = CompressType::SNAPPYBLOCK;
- break;
- default:
- return Status::InternalError<false>("unknown compress type: {}",
_file_compress_type);
- }
+ RETURN_IF_ERROR(Decompressor::create_decompressor(_file_compress_type,
&decompressor));
Review Comment:
> pls add test cases for this feature
I have added test cases into the **suite("test_compress_type", "load_p0")**.
However, these cases depend on three files stored in external object storage,
and I need your help to upload these three files to the object storage. I have
already included these three files in my commit, and they are:
- regression-test/data/load_p0/stream_load/basic_data_by_line.json.gz
- regression-test/data/load_p0/stream_load/basic_data_by_line.json.lz4
- regression-test/data/load_p0/stream_load/basic_data_by_line.json.bz2
I hope you can upload them to the following path: @xiaokang
- s3://doris-build-1308700295/regression/load/data/basic_data_by_line.json.gz
-
s3://doris-build-1308700295/regression/load/data/basic_data_by_line.json.lz4
-
s3://doris-build-1308700295/regression/load/data/basic_data_by_line.json.bz2
> It will cause memory leak if error occures and return without delete
decompressor. You can change arg of create_decompressor from Decompressor** to
unique_ptr*
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
