This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dea1b99e528 [fix](hive) Match OpenCSVSerde semantics in FileScannerV2
(#68027)
dea1b99e528 is described below
commit dea1b99e528f57b96d9b3bd9fa68f4dc3e726d32
Author: Gabriel <[email protected]>
AuthorDate: Thu Sep 17 10:29:42 2026 +0800
[fix](hive) Match OpenCSVSerde semantics in FileScannerV2 (#68027)
### What problem does this PR solve?
Hive CSV tables created by Trino can store character settings only in
table parameters. Doris used SerDe parameters and a generic CSV parser,
producing merged, shifted, or empty data columns even when partition
values were correct.
Resolve table-over-SerDe character precedence, first-Java-character
normalization, defaults, and Hive's double-quote escape sentinel.
Validate the effective character tuple and reject settings that cannot
be represented by the existing character fields.
Implement OpenCSV row semantics in FileScannerV2. The parser and
physical-record context live under `be/src/format_v2/delimited_text/`,
with tests under `be/test/format_v2/delimited_text/`. The legacy
`be/src/format/` directory is unchanged from the PR base. Match OpenCSV
2.3 field transitions, including leading whitespace, embedded and
doubled quotes, disabled quote/escape settings, and literal NUL bytes.
Preserve Hadoop physical records (LF, CRLF, and CR), completed fields
before an unmatched quote, missing-field NULLs, explicit empty strings,
and file-start BOM handling. Decode each field once and bound delimiter
metadata by the requested column prefix. Neither table nor SerDe
`line.delim` overrides CSV input framing.
Version the contract at both boundaries: connector API 9.0 freezes the
public scan-property names, types, and values; OpenCSV scans require
query-wide BE execution version 15 or newer and exclude smooth-upgrade
source backends. Requests without the optional semantic flag retain
generic CSV decoding in FileScannerV2.
The external regression uses Hive-compatible, non-NUL character
properties because PostgreSQL-backed Hive metastores cannot store NUL in
text metadata. Disabled quote/escape combinations remain covered by the
Hive oracle and V2 unit tests; raw file records still include binary
NULs.
### Release note
Fix incorrect results when FileScannerV2 reads Hive OpenCSV tables with
custom character properties and OpenCSV-specific quoting or escaping.
Reject unsupported character metadata explicitly. OpenCSV scans require
BE execution version 15 or newer; connector plugins must use API 9.0.
### Check List (For Author)
- Test:
- 480 Hive connector tests passed, including an actual Hive 3.1.3 oracle
for 343 character tuples and 606 record cases with deterministic random
inputs and binary NULs.
- 142 connector SPI tests passed, including the API 9.0 pin and
regenerated surface baselines. The metadata-method baseline is
unchanged.
- Prior FE Core validation passed 34 tests: 13 CSV wire/upgrade tests,
13 plugin loading/version tests, and eight existing scan compatibility
tests. Coverage includes an API 8.0 plugin jar, execution versions
14/15, smooth-upgrade source rejection, and absent-field compatibility.
This update does not modify those implementations.
- 43 BE tests passed under ASAN. FileScannerV2 consumes the
Hive-generated corpus across batches, projections, physical line
endings, counts, split offsets, and BOMs, including disabled character
settings and absent-flag generic decoding.
- External regression explicitly enables FileScannerV2 and covers
partitioned/unpartitioned layouts, table/SerDe properties, raw TEXTFILE
records, projections, filters, counts, and metadata-only changes. Groovy
compilation passed; the end-to-end external rerun remains pending CI.
- A separate probe used Hive 3.1.3 SQL literal decoding and a real
PostgreSQL 17 instance: the original fixture fails on NUL metadata; all
12 property values in the four revised dialects persist successfully.
- FE Checkstyle, clang-format 16 on all affected C++ files, header
hygiene, License Eye, and whitespace checks passed. Repository
clang-tidy remains blocked by the existing unmatched `NOLINTEND` in
`core/types.h` and existing reader diagnostics. The V2 parser and
physical-record helper have no clang-tidy diagnostics.
- Behavior changed: Yes, Hive CSV property resolution and FileScannerV2
row semantics now match OpenCSVSerde for the supported character
settings.
- Does this need documentation: No.
---
.licenserc.yaml | 4 +
be/src/agent/be_exec_version_manager.cpp | 4 +-
be/src/agent/be_exec_version_manager.h | 1 +
be/src/format_v2/delimited_text/csv_reader.cpp | 33 +-
be/src/format_v2/delimited_text/csv_reader.h | 4 +
.../delimited_text/hive_csv_line_reader.cpp | 40 ++
.../delimited_text/hive_csv_line_reader.h | 34 ++
.../format_v2/delimited_text/hive_csv_parser.cpp | 120 ++++
be/src/format_v2/delimited_text/hive_csv_parser.h | 45 ++
.../delimited_text/hive_csv_reader_test.cpp | 310 +++++++++++
.../main/java/org/apache/doris/common/Config.java | 4 +-
fe/fe-connector/fe-connector-hive/pom.xml | 15 +
.../doris/connector/hive/HiveTextProperties.java | 81 ++-
.../hive/HiveCsvReaderCompatibilityTest.java | 161 ++++++
.../connector/hive/HiveScanBatchModeTest.java | 25 +
.../connector/hive/HiveTextPropertiesTest.java | 169 ++++++
.../src/test/resources/hive-csv-oracle.tsv | 606 +++++++++++++++++++++
.../connector/spi/scan/ScanNodePropertyKeys.java | 3 +
.../connector/spi/ConnectorPluginSurfaceTest.java | 23 +-
.../test/resources/connector-plugin-surface.txt | 24 +
fe/fe-connector/pom.xml | 2 +-
.../datasource/scan/PluginDrivenScanNode.java | 8 +
.../PluginDrivenScanNodeCsvPropertiesTest.java | 224 ++++++++
.../PluginApiVersionWiringTest.java | 8 +
gensrc/thrift/PlanNodes.thrift | 4 +
.../hive/test_csv_table_properties.groovy | 266 +++++++++
26 files changed, 2181 insertions(+), 37 deletions(-)
diff --git a/.licenserc.yaml b/.licenserc.yaml
index 7282390c0d1..92608eadef1 100644
--- a/.licenserc.yaml
+++ b/.licenserc.yaml
@@ -65,6 +65,10 @@ header:
# the whole file and compares it to this one line for line, so a header
would read back
# as extra rows; it prints the regenerated path on mismatch instead.
- "fe/fe-core/src/test/resources/access-control-behavior-baseline.txt"
+ # Hive-generated CSV oracle. The FE test compares every line with
regenerated
+ # data, and both BE readers consume every line as a record; a header would
+ # become unexpected test data. Exclude only this fixture, not all TSV
files.
+ -
"fe/fe-connector/fe-connector-hive/src/test/resources/hive-csv-oracle.tsv"
# Golden protocol traffic. MysqlPacketGoldenTest and
FlightResultGoldenTest regenerate
# each file in full and compare it to this one byte for byte, so a header
would read back
# as unexpected leading content; both tests print the regeneration command
on mismatch.
diff --git a/be/src/agent/be_exec_version_manager.cpp
b/be/src/agent/be_exec_version_manager.cpp
index aed123eca87..ded7a31240a 100644
--- a/be/src/agent/be_exec_version_manager.cpp
+++ b/be/src/agent/be_exec_version_manager.cpp
@@ -133,8 +133,10 @@ void
BeExecVersionManager::check_function_compatibility(int current_be_exec_vers
// b. support Paimon default fixed-bucket routing in the external sink
exchange.
// 14: start from master
// a. support TIMESTAMP_NS in Thrift descriptors and PBlock exchange.
+// 15: start from master
+// a. distinguish Hive OpenCSVSerde row semantics from generic CSV decoding
during upgrades.
-const int BeExecVersionManager::max_be_exec_version =
SUPPORT_TIMESTAMP_NS_VERSION;
+const int BeExecVersionManager::max_be_exec_version =
SUPPORT_HIVE_OPEN_CSV_VERSION;
const int BeExecVersionManager::min_be_exec_version = 0;
std::map<std::string, std::set<int>>
BeExecVersionManager::_function_change_map {};
std::set<std::string> BeExecVersionManager::_function_restrict_map;
diff --git a/be/src/agent/be_exec_version_manager.h
b/be/src/agent/be_exec_version_manager.h
index 32a317786a5..4a7ca9de6e4 100644
--- a/be/src/agent/be_exec_version_manager.h
+++ b/be/src/agent/be_exec_version_manager.h
@@ -34,6 +34,7 @@ constexpr inline int
SUPPORT_ICEBERG_MERGE_CARDINALITY_VERSION = 11;
constexpr inline int SUPPORT_ICEBERG_VARIANT_VERSION = 12;
constexpr inline int SUPPORT_EXTERNAL_TABLE_SINK_HASH_VERSION = 13;
constexpr inline int SUPPORT_TIMESTAMP_NS_VERSION = 14;
+constexpr inline int SUPPORT_HIVE_OPEN_CSV_VERSION = 15;
class BeExecVersionManager {
public:
diff --git a/be/src/format_v2/delimited_text/csv_reader.cpp
b/be/src/format_v2/delimited_text/csv_reader.cpp
index bb6c55d3084..5bfd57346b9 100644
--- a/be/src/format_v2/delimited_text/csv_reader.cpp
+++ b/be/src/format_v2/delimited_text/csv_reader.cpp
@@ -26,6 +26,8 @@
#include "core/data_type_serde/data_type_string_serde.h"
#include "format/file_reader/new_plain_binary_line_reader.h"
#include "format/file_reader/new_plain_text_line_reader.h"
+#include "format_v2/delimited_text/hive_csv_line_reader.h"
+#include "format_v2/delimited_text/hive_csv_parser.h"
#include "gen_cpp/internal_service.pb.h"
#include "runtime/descriptors.h"
#include "runtime/runtime_state.h"
@@ -117,6 +119,17 @@ Status CsvReader::_init_format_state() {
if (text_params.__isset.empty_field_as_null) {
_empty_field_as_null = text_params.empty_field_as_null;
}
+ if (_scan_params->file_attributes.hive_open_csv) {
+ size_t fields = _source_file_slot_descs.size();
+ for (int index : _scan_params->column_idxs) {
+ fields = std::max(fields, static_cast<size_t>(index) + 1);
+ }
+ _hive_csv_parser =
+ std::make_unique<HiveCsvParser>(_value_separator, _enclose,
_escape, fields);
+ _options.escape_char = 0;
+ _options.converted_from_string = false;
+ _options.null_len = 0;
+ }
return Status::OK();
}
@@ -130,7 +143,9 @@ Status CsvReader::_create_decompressor() {
Status CsvReader::_create_line_reader() {
if (is_csv_text_format(_file_format_type)) {
std::shared_ptr<TextLineReaderContextIf> text_line_reader_ctx;
- if (_enclose == 0) {
+ if (_hive_csv_parser) {
+ text_line_reader_ctx = std::make_shared<HiveCsvLineReaderCtx>();
+ } else if (_enclose == 0) {
text_line_reader_ctx = std::make_shared<PlainTextLineReaderCtx>(
_line_delimiter, _line_delimiter.size(), _keep_cr);
} else {
@@ -164,6 +179,10 @@ Status CsvReader::_validate_line(const Slice& line) {
void CsvReader::_split_line(const Slice& line) {
_split_values.clear();
+ if (_hive_csv_parser) {
+ _hive_csv_parser->parse(line, &_split_values);
+ return;
+ }
if (_file_format_type == TFileFormatType::FORMAT_PROTO) {
auto** row_ptr = reinterpret_cast<PDataRow**>(line.data);
PDataRow* row = *row_ptr;
@@ -212,10 +231,20 @@ void CsvReader::_split_line(const Slice& line) {
Status CsvReader::_deserialize_one_cell(const RequestedColumn& column,
IColumn* output,
Slice value) {
DORIS_CHECK(output != nullptr);
+ if (_hive_csv_parser && column.file_column_id.value() >=
_split_values.size()) {
+ return _append_null(output);
+ }
if (column.nullable_string_fast_path) {
auto& null_column = assert_cast<ColumnNullable&>(*output);
// String is the hottest CSV type. Avoid the generic nullable serde
wrapper here:
// deserialize directly into the nested string column and append the
null map bit ourselves.
+ if (_hive_csv_parser) {
+ // Splitting and decoding are one operation for OpenCSV; a second
CSV serde pass
+ // would unescape literal data and confuse empty strings with
missing fields.
+ null_column.get_nested_column().insert_data(value.data,
value.size);
+ null_column.get_null_map_data().push_back(0);
+ return Status::OK();
+ }
if (_empty_field_as_null && value.size == 0) {
null_column.insert_data(nullptr, 0);
return Status::OK();
@@ -243,7 +272,7 @@ Status CsvReader::_deserialize_one_cell(const
RequestedColumn& column, IColumn*
}
Slice CsvReader::_normalize_value(Slice value) const {
- if (_empty_field_as_null && value.size == 0) {
+ if (!_hive_csv_parser && _empty_field_as_null && value.size == 0) {
return Slice(_options.null_format, _options.null_len);
}
return value;
diff --git a/be/src/format_v2/delimited_text/csv_reader.h
b/be/src/format_v2/delimited_text/csv_reader.h
index dd1d77d5a34..c8565e6792a 100644
--- a/be/src/format_v2/delimited_text/csv_reader.h
+++ b/be/src/format_v2/delimited_text/csv_reader.h
@@ -31,6 +31,8 @@ class SlotDescriptor;
namespace doris::format::csv {
+class HiveCsvParser;
+
// FileScannerV2 CSV reader.
//
// CSV files do not carry a physical schema. FE provides the table slot
descriptors plus
@@ -62,9 +64,11 @@ private:
Slice value) override;
Slice _normalize_value(Slice value) const override;
bool _can_split() const override;
+ bool _empty_line_as_record() const override { return _hive_csv_parser !=
nullptr; }
void _on_bom_removed(size_t bom_size) override;
TFileFormatType::type _file_format_type =
TFileFormatType::FORMAT_CSV_PLAIN;
+ std::unique_ptr<HiveCsvParser> _hive_csv_parser;
char _enclose = 0;
bool _trim_double_quotes = false;
bool _trim_tailing_spaces = false;
diff --git a/be/src/format_v2/delimited_text/hive_csv_line_reader.cpp
b/be/src/format_v2/delimited_text/hive_csv_line_reader.cpp
new file mode 100644
index 00000000000..dcfc1da27b8
--- /dev/null
+++ b/be/src/format_v2/delimited_text/hive_csv_line_reader.cpp
@@ -0,0 +1,40 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/delimited_text/hive_csv_line_reader.h"
+
+namespace doris::format::csv {
+
+const uint8_t* HiveCsvLineReaderCtx::read_line(const uint8_t* start, size_t
len) {
+ for (size_t i = 0; i < len; ++i) {
+ if (start[i] == '\n') {
+ _delimiter_length = 1;
+ return start + i;
+ }
+ if (start[i] == '\r') {
+ // Wait for lookahead when CR straddles input buffers. At EOF the
parser removes it.
+ if (i + 1 == len) {
+ return nullptr;
+ }
+ _delimiter_length = start[i + 1] == '\n' ? 2 : 1;
+ return start + i;
+ }
+ }
+ return nullptr;
+}
+
+} // namespace doris::format::csv
diff --git a/be/src/format_v2/delimited_text/hive_csv_line_reader.h
b/be/src/format_v2/delimited_text/hive_csv_line_reader.h
new file mode 100644
index 00000000000..7754fd31fb8
--- /dev/null
+++ b/be/src/format_v2/delimited_text/hive_csv_line_reader.h
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "format/file_reader/new_plain_text_line_reader.h"
+
+namespace doris::format::csv {
+
+class HiveCsvLineReaderCtx final : public TextLineReaderContextIf {
+public:
+ const uint8_t* read_line(const uint8_t* start, size_t len) override;
+ size_t line_delimiter_length() const override { return _delimiter_length; }
+ void refresh() override {}
+
+private:
+ size_t _delimiter_length = 1;
+};
+
+} // namespace doris::format::csv
diff --git a/be/src/format_v2/delimited_text/hive_csv_parser.cpp
b/be/src/format_v2/delimited_text/hive_csv_parser.cpp
new file mode 100644
index 00000000000..9ebaef67477
--- /dev/null
+++ b/be/src/format_v2/delimited_text/hive_csv_parser.cpp
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format_v2/delimited_text/hive_csv_parser.h"
+
+#include <unicode/uchar.h>
+#include <unicode/utf8.h>
+
+#include <string_view>
+#include <utility>
+
+namespace doris::format::csv {
+namespace {
+
+UChar32 next_character(std::string_view input, size_t& pos) {
+ UChar32 character;
+ U8_NEXT(input, pos, input.size(), character);
+ return character;
+}
+
+} // namespace
+
+HiveCsvParser::HiveCsvParser(std::string separator, char quote, char escape,
size_t field_limit)
+ : _separator(std::move(separator)),
+ _quote(quote),
+ _escape(escape),
+ _field_limit(field_limit) {}
+
+void HiveCsvParser::parse(const Slice& line, std::vector<Slice>* fields) {
+ fields->clear();
+ _decoded.clear();
+ _field_ends.clear();
+ std::string_view input(line.data, line.size);
+ // A terminal CR may reach here at EOF, before the line reader can look
ahead for CRLF.
+ if (!input.empty() && input.back() == '\r') {
+ input.remove_suffix(1);
+ }
+ if (input.empty()) {
+ return;
+ }
+ bool in_quotes = false;
+ bool in_field = false;
+ bool all_whitespace = true;
+ bool previous_separator = false;
+ size_t field_start = 0;
+ size_t java_position = 0;
+ for (size_t pos = 0; pos < input.size() && _field_ends.size() <
_field_limit;) {
+ size_t start = pos;
+ const UChar32 character = next_character(input, pos);
+ bool separator = input.substr(start, pos - start) == _separator;
+ if (character == _escape) {
+ // OpenCSV checks the escape branch even for NUL. An escape
outside a field is
+ // discarded, while quotes/escapes inside a field consume their
following byte.
+ if ((in_quotes || in_field) && pos < input.size() &&
+ (input[pos] == _quote || input[pos] == _escape)) {
+ _decoded.push_back(input[pos++]);
+ all_whitespace = all_whitespace && u_isWhitespace(input[pos -
1]);
+ ++java_position;
+ }
+ } else if (character == _quote) {
+ if ((in_quotes || in_field) && pos < input.size() && input[pos] ==
_quote) {
+ _decoded.push_back(input[pos++]);
+ all_whitespace = all_whitespace && u_isWhitespace(_quote);
+ ++java_position;
+ } else {
+ // OpenCSV's embedded-quote rule uses the UTF-16 position in
the entire record,
+ // not the field offset. Quotes still toggle state after a
non-whitespace prefix.
+ if (java_position > 2 && !previous_separator && pos <
input.size() &&
+ !input.substr(pos).starts_with(_separator)) {
+ if (_decoded.size() > field_start && all_whitespace) {
+ _decoded.resize(field_start);
+ } else {
+ _decoded.push_back(_quote);
+ all_whitespace = all_whitespace &&
u_isWhitespace(_quote);
+ }
+ }
+ in_quotes = !in_quotes;
+ }
+ in_field = !in_field;
+ } else if (separator && !in_quotes) {
+ _field_ends.push_back(_decoded.size());
+ field_start = _decoded.size();
+ all_whitespace = true;
+ in_field = false;
+ } else {
+ _decoded.append(input.substr(start, pos - start));
+ all_whitespace = all_whitespace && u_isWhitespace(character);
+ in_field = true;
+ }
+ // Escaped lookahead cannot be a separator: FE validates distinct
active characters.
+ previous_separator = separator;
+ java_position += character > 0xffff ? 2 : 1;
+ }
+ // CSVReader.readNext() returns completed fields at EOF and drops only the
pending field.
+ // Do not merge the next physical record or discard fields completed
before an unmatched quote.
+ if (!in_quotes && _field_ends.size() < _field_limit) {
+ _field_ends.push_back(_decoded.size());
+ }
+ size_t start = 0;
+ for (size_t end : _field_ends) {
+ fields->emplace_back(_decoded.data() + start, end - start);
+ start = end;
+ }
+}
+
+} // namespace doris::format::csv
diff --git a/be/src/format_v2/delimited_text/hive_csv_parser.h
b/be/src/format_v2/delimited_text/hive_csv_parser.h
new file mode 100644
index 00000000000..2984dae485f
--- /dev/null
+++ b/be/src/format_v2/delimited_text/hive_csv_parser.h
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <string>
+#include <vector>
+
+#include "util/slice.h"
+
+namespace doris::format::csv {
+
+// OpenCSVSerde runs one OpenCSV 2.3 reader per Hadoop TextInputFormat record.
+// Returned slices reference this parser's decoded buffer until the next
parse().
+// The field limit bounds delimiter metadata by the requested column prefix.
+class HiveCsvParser {
+public:
+ HiveCsvParser(std::string separator, char quote, char escape, size_t
field_limit);
+ void parse(const Slice& line, std::vector<Slice>* fields);
+
+private:
+ std::string _separator;
+ char _quote;
+ char _escape;
+ size_t _field_limit;
+ std::string _decoded;
+ std::vector<size_t> _field_ends;
+};
+
+} // namespace doris::format::csv
diff --git a/be/test/format_v2/delimited_text/hive_csv_reader_test.cpp
b/be/test/format_v2/delimited_text/hive_csv_reader_test.cpp
new file mode 100644
index 00000000000..cc4eb8e03fb
--- /dev/null
+++ b/be/test/format_v2/delimited_text/hive_csv_reader_test.cpp
@@ -0,0 +1,310 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <array>
+#include <cstdlib>
+#include <filesystem>
+#include <fstream>
+#include <map>
+#include <optional>
+#include <sstream>
+
+#include "agent/be_exec_version_manager.h"
+#include "common/object_pool.h"
+#include "core/block/block.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "format_v2/delimited_text/csv_reader.h"
+#include "format_v2/delimited_text/hive_csv_line_reader.h"
+#include "format_v2/delimited_text/hive_csv_parser.h"
+#include "io/io_common.h"
+#include "testutil/desc_tbl_builder.h"
+#include "testutil/mock/mock_runtime_state.h"
+
+namespace doris::format::csv {
+namespace {
+
+using OracleRow = std::array<std::optional<std::string>, 3>;
+struct OracleRecord {
+ std::string input;
+ OracleRow expected;
+};
+
+std::string unhex(const std::string& hex) {
+ if (hex == "EMPTY") {
+ return "";
+ }
+ std::string value;
+ for (size_t i = 0; i < hex.size(); i += 2) {
+ value.push_back(static_cast<char>(std::stoi(hex.substr(i, 2), nullptr,
16)));
+ }
+ return value;
+}
+
+std::map<std::string, std::vector<OracleRecord>> read_oracle() {
+ // HiveCsvReaderCompatibilityTest regenerates and verifies this shared
corpus with Hive 3.1.3.
+ const char* root = std::getenv("ROOT");
+ if (root == nullptr) {
+ ADD_FAILURE() << "Set ROOT to the repository root, as run-be-ut.sh
does";
+ return {};
+ }
+ std::ifstream input(
+ std::string(root) +
+
"/fe/fe-connector/fe-connector-hive/src/test/resources/hive-csv-oracle.tsv");
+ EXPECT_TRUE(input.is_open());
+ std::map<std::string, std::vector<OracleRecord>> corpus;
+ std::string line;
+ while (std::getline(input, line)) {
+ std::istringstream fields(line);
+ std::string tuple;
+ std::string raw;
+ std::getline(fields, tuple, '\t');
+ std::getline(fields, raw, '\t');
+ OracleRecord record {.input = unhex(raw), .expected = {}};
+ for (auto& value : record.expected) {
+ std::string encoded;
+ std::getline(fields, encoded, '\t');
+ if (encoded != "NULL") {
+ value = unhex(encoded);
+ }
+ }
+ corpus[unhex(tuple)].push_back(std::move(record));
+ }
+ EXPECT_EQ(corpus.size(), 6);
+ return corpus;
+}
+
+TEST(HiveCsvParserTest, HiveRecordOracle) {
+ for (const auto& [tuple, records] : read_oracle()) {
+ std::string separator = tuple.substr(0, tuple.size() - 2);
+ for (size_t limit : {1, 2, 3}) {
+ HiveCsvParser parser(separator, tuple[tuple.size() - 2],
tuple.back(), limit);
+ for (const auto& record : records) {
+ std::vector<Slice> fields;
+ parser.parse(Slice(record.input), &fields);
+ ASSERT_LE(fields.size(), limit);
+ for (size_t i = 0; i < limit; ++i) {
+ std::optional<std::string> actual;
+ if (i < fields.size()) {
+ actual = fields[i].to_string();
+ }
+ EXPECT_EQ(actual, record.expected[i]) << "field " << i;
+ }
+ }
+ }
+ }
+}
+
+TEST(HiveCsvCompatibilityTest, AdvertisesOpenCsvExecutionVersion) {
+ // Version 15 identifies the backend generation that implements the
OpenCSV semantic flag.
+ EXPECT_GE(BeExecVersionManager::get_newest_version(), 15);
+ EXPECT_TRUE(BeExecVersionManager::check_be_exec_version(15).ok());
+ EXPECT_TRUE(BeExecVersionManager::check_be_exec_version(14).ok());
+}
+
+TEST(HiveCsvLineReaderTest, CrLfAcrossBuffersAndQuotes) {
+ HiveCsvLineReaderCtx context;
+ const std::string input = "qleft\r\nrightq|tail\n";
+ const auto* data = reinterpret_cast<const uint8_t*>(input.data());
+ EXPECT_EQ(context.read_line(data, 6), nullptr);
+ EXPECT_EQ(context.read_line(data, input.size()), data + 5);
+ EXPECT_EQ(context.line_delimiter_length(), 2);
+ context.refresh();
+ EXPECT_EQ(context.read_line(data + 7, input.size() - 7), data +
input.size() - 1);
+ EXPECT_EQ(context.line_delimiter_length(), 1);
+ const std::string cr = "left\rright";
+ data = reinterpret_cast<const uint8_t*>(cr.data());
+ EXPECT_EQ(context.read_line(data, cr.size()), data + 4);
+ EXPECT_EQ(context.line_delimiter_length(), 1);
+}
+
+TFileScanRangeParams hive_csv_params(const std::string& tuple, const
std::vector<int>& projection,
+ const std::vector<SlotDescriptor*>& slots,
+ bool hive_open_csv = true) {
+ TFileScanRangeParams params;
+ params.__set_format_type(TFileFormatType::FORMAT_CSV_PLAIN);
+ params.__set_file_type(TFileType::FILE_LOCAL);
+ params.__set_compress_type(TFileCompressType::PLAIN);
+ params.__set_column_idxs(projection);
+ if (hive_open_csv) {
+ params.file_attributes.__set_hive_open_csv(true);
+ } else {
+ // Thrift marks explicit defaults as set on construction; model an
older sender omitting the field.
+ params.file_attributes.__isset.hive_open_csv = false;
+ }
+ params.file_attributes.__set_header_type("");
+ params.file_attributes.__set_trim_double_quotes(true);
+ auto& text = params.file_attributes.text_params;
+ text.__set_column_separator(tuple.substr(0, tuple.size() - 2));
+ text.__set_line_delimiter("\n");
+ text.__set_enclose(tuple[tuple.size() - 2]);
+ text.__set_escape(tuple.back());
+ text.__set_null_format("");
+ text.__set_empty_field_as_null(false);
+ params.file_attributes.__isset.text_params = true;
+ params.__isset.file_attributes = true;
+ for (auto* slot : slots) {
+ TFileScanSlotInfo info;
+ info.__set_slot_id(slot->id());
+ info.__set_is_file_slot(true);
+ params.required_slots.push_back(info);
+ }
+ return params;
+}
+
+void check_reader_rows(CsvReader* reader, const std::vector<SlotDescriptor*>&
slots,
+ const DataTypePtr& type, const
std::vector<OracleRecord>& records,
+ const std::vector<int>& projection, size_t
skipped_rows) {
+ size_t row_index = skipped_rows;
+ bool eof = false;
+ while (!eof) {
+ Block block;
+ for (auto* slot : slots) {
+ block.insert({type->create_column(), type, slot->col_name()});
+ }
+ size_t rows = 0;
+ auto status = reader->get_block(&block, &rows, &eof);
+ ASSERT_TRUE(status.ok()) << status;
+ for (size_t row = 0; row < rows; ++row, ++row_index) {
+ ASSERT_LT(row_index, records.size());
+ for (size_t col = 0; col < projection.size(); ++col) {
+ const auto& column =
+ static_cast<const
ColumnNullable&>(*block.get_by_position(col).column);
+ std::optional<std::string> actual;
+ if (!column.is_null_at(row)) {
+ actual =
column.get_nested_column().get_data_at(row).to_string();
+ }
+ EXPECT_EQ(actual, records[row_index].expected[projection[col]])
+ << "record " << row_index << " column " <<
projection[col];
+ }
+ }
+ }
+ EXPECT_EQ(row_index, records.size());
+}
+
+void check_reader_count(CsvReader* reader, size_t expected) {
+ format::FileAggregateRequest request;
+ request.agg_type = TPushAggOp::COUNT;
+ format::FileAggregateResult result;
+ ASSERT_TRUE(reader->get_aggregate_result(request, &result).ok());
+ EXPECT_EQ(result.count, expected);
+}
+
+class HiveOpenCsvReaderTest : public testing::Test {
+protected:
+ void SetUp() override {
+ _directory = std::filesystem::temp_directory_path() /
"hive_csv_reader_v2";
+ std::filesystem::create_directories(_directory);
+ }
+ void TearDown() override { std::filesystem::remove_all(_directory); }
+
+ void check_file(const std::string& tuple, const std::vector<OracleRecord>&
records,
+ const std::string& delimiter, const std::vector<int>&
projection,
+ int64_t start_offset = 0, size_t skipped_rows = 0, bool
count = false,
+ bool hive_open_csv = true) {
+ const auto path = (_directory / "records.csv").string();
+ {
+ std::ofstream output(path, std::ios::binary);
+ for (const auto& record : records) {
+ output << record.input << delimiter;
+ }
+ }
+ MockRuntimeState state;
+ // Tiny batches exercise parser-buffer lifetime and state reset
between blocks.
+ state._query_options.__set_batch_size(7);
+ state._query_options.__set_keep_carriage_return(true);
+ RuntimeProfile profile("hive_csv_reader");
+ ObjectPool pool;
+ DescriptorTblBuilder builder(&pool);
+ auto& tuple_builder = builder.declare_tuple();
+ auto type = make_nullable(std::make_shared<DataTypeString>());
+ for (int index : projection) {
+ tuple_builder << TupleDescBuilder::SlotType {type, "c" +
std::to_string(index)};
+ }
+ auto slots = builder.build()->get_tuple_descriptor(0)->slots();
+ auto params = hive_csv_params(tuple, projection, slots, hive_open_csv);
+ ASSERT_EQ(params.file_attributes.__isset.hive_open_csv, hive_open_csv);
+ const auto file_size = std::filesystem::file_size(path);
+ auto properties = std::make_shared<io::FileSystemProperties>();
+ properties->system_type = TFileType::FILE_LOCAL;
+ auto description = std::make_unique<io::FileDescription>();
+ description->path = path;
+ description->range_start_offset = start_offset;
+ description->range_size = file_size - start_offset;
+ description->file_size = file_size;
+ CsvReader reader(properties, description, nullptr, &profile, ¶ms,
slots);
+ ASSERT_TRUE(reader.init(&state).ok());
+ auto request = std::make_shared<format::FileScanRequest>();
+ for (size_t i = 0; i < projection.size(); ++i) {
+ format::LocalColumnId id(projection[i]);
+
request->non_predicate_columns.push_back(format::LocalColumnIndex::top_level(id));
+ request->local_positions.emplace(id, format::LocalIndex(i));
+ }
+ ASSERT_TRUE(reader.open(request).ok());
+ if (count) {
+ check_reader_count(&reader, records.size() - skipped_rows);
+ } else {
+ check_reader_rows(&reader, slots, type, records, projection,
skipped_rows);
+ }
+ ASSERT_TRUE(reader.close().ok());
+ }
+
+ std::filesystem::path _directory;
+};
+
+TEST_F(HiveOpenCsvReaderTest, HiveRecordOracleAcrossBatchesAndProjections) {
+ for (const auto& [tuple, records] : read_oracle()) {
+ for (const std::string delimiter : {"\n", "\r\n", "\r"}) {
+ check_file(tuple, records, delimiter, {0, 1, 2});
+ check_file(tuple, records, delimiter, {2, 0});
+ check_file(tuple, records, delimiter, {0, 1, 2}, 0, 0, true);
+ }
+ }
+}
+
+TEST_F(HiveOpenCsvReaderTest,
UnterminatedQuotesDoNotJoinPhysicalRecordsOrSplits) {
+ std::vector<OracleRecord> records = {
+ {.input = "qleft", .expected = {}},
+ {.input = "rightq|tail", .expected = {}},
+ {.input = "left|qunclosed", .expected = {"lft", {}, {}}},
+ {.input = "abcqleft|rightq|tail", .expected = {"abcqlft|right",
"tail", {}}}};
+ check_file("|qe", records, "\n", {0, 1, 2});
+ check_file("|qe", records, "\n", {2, 0}, 2, 1);
+}
+
+TEST_F(HiveOpenCsvReaderTest, BomIsOnlyStrippedAtFileStart) {
+ std::vector<OracleRecord> records(9, {.input = "plain", .expected =
{"plain", {}, {}}});
+ records[0].input = "\xef\xbb\xbfplain";
+ records[7] = {.input = "\xef\xbb\xbfplain", .expected =
{"\xef\xbb\xbfplain", {}, {}}};
+ check_file("|qe", records, "\n", {0, 1, 2});
+}
+
+TEST_F(HiveOpenCsvReaderTest, AbsentSemanticFlagRetainsLegacyDecoding) {
+ // The same bytes distinguish a pre-flag request from the OpenCSV opt-in
on a new backend.
+ std::string tuple(",\0e", 3);
+ std::vector<OracleRecord> records = {
+ {.input = "eeabc,tail,last", .expected = {"eabc", "tail",
"last"}}};
+ check_file(tuple, records, "\n", {0, 1, 2}, 0, 0, false, false);
+ records[0].expected[0] = "abc";
+ check_file(tuple, records, "\n", {0, 1, 2});
+}
+
+} // namespace
+} // namespace doris::format::csv
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 9cf5ac9d09b..3a4b757f3c2 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2008,9 +2008,11 @@ public class Config extends ConfigBase {
* Max data version of backends serialize block.
*/
public static final int TIMESTAMP_NS_MIN_BE_EXEC_VERSION = 14;
+ // Older backends ignore the optional OpenCSV flag and would silently use
different row semantics.
+ public static final int HIVE_OPEN_CSV_MIN_BE_EXEC_VERSION = 15;
@ConfField(mutable = false)
- public static int max_be_exec_version = TIMESTAMP_NS_MIN_BE_EXEC_VERSION;
+ public static int max_be_exec_version = HIVE_OPEN_CSV_MIN_BE_EXEC_VERSION;
/**
* Min data version of backends serialize block.
diff --git a/fe/fe-connector/fe-connector-hive/pom.xml
b/fe/fe-connector/fe-connector-hive/pom.xml
index 2152c990f53..4cb30703c3f 100644
--- a/fe/fe-connector/fe-connector-hive/pom.xml
+++ b/fe/fe-connector/fe-connector-hive/pom.xml
@@ -114,6 +114,14 @@ under the License.
<artifactId>log4j-api</artifactId>
</dependency>
+ <!-- Match the parser used by Hive 3.1.3 for reader-compatibility
tests only. -->
+ <dependency>
+ <groupId>net.sf.opencsv</groupId>
+ <artifactId>opencsv</artifactId>
+ <version>2.3</version>
+ <scope>test</scope>
+ </dependency>
+
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
@@ -146,6 +154,13 @@ under the License.
<build>
<finalName>doris-fe-connector-hive</finalName>
<plugins>
+ <plugin>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <!-- Hive 3's real SerDe initializes StringInternUtils,
which reflects on URI on Java 17. -->
+ <argLine>@{argLine}
--add-opens=java.base/java.net=ALL-UNNAMED</argLine>
+ </configuration>
+ </plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
diff --git
a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java
b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java
index bcbaaeb535b..498ff5a8c27 100644
---
a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java
+++
b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java
@@ -17,6 +17,7 @@
package org.apache.doris.connector.hive;
+import org.apache.doris.connector.spi.DorisConnectorException;
import org.apache.doris.connector.spi.scan.ScanNodePropertyKeys;
import java.util.HashMap;
@@ -37,8 +38,8 @@ import java.util.Map;
* {@link HiveCatalogProperties} (per catalog) and {@link HmsConf} (per FE).
Its input keys are remote HMS
* table parameters and its output keys are a BE payload, so both sets belong
to the single class that
* reads them, which is this one. That is also why its two numeric parses stay
lenient: a garbage
- * {@code skip.header.line.count} or delimiter in somebody else's Hive table
must not fail the query the
- * way a garbage value a user typed into {@code CREATE CATALOG} now does.</p>
+ * {@code skip.header.line.count} or LazySimpleSerDe delimiter keeps its
historical fallback. CSV
+ * character properties instead follow OpenCSVSerde validation and the BE
wire-format limits.</p>
*/
public final class HiveTextProperties {
@@ -103,7 +104,7 @@ public final class HiveTextProperties {
*
* @param serDeLib the SerDe library class name
* @param sdParams the StorageDescriptor / SerDeInfo parameters
- * @param tableParams the table-level parameters (for
skip.header.line.count)
+ * @param tableParams the table-level SerDe properties and
skip.header.line.count
* @return map of text properties, empty if not a text-based format
*/
public static Map<String, String> extract(String serDeLib,
@@ -117,7 +118,7 @@ public final class HiveTextProperties {
if (HIVE_TEXT_SERDE.equals(serDeLib) || multiDelimit) {
extractTextSerDeProps(sdParams, tableParams, result, multiDelimit);
} else if (HIVE_OPEN_CSV_SERDE.equals(serDeLib)) {
- extractCsvSerDeProps(sdParams, result);
+ extractCsvSerDeProps(sdParams, tableParams, result);
} else if (HIVE_JSON_SERDE.equals(serDeLib) ||
LEGACY_HIVE_JSON_SERDE.equals(serDeLib)
|| OPENX_JSON_SERDE.equals(serDeLib)) {
extractJsonSerDeProps(serDeLib, sdParams, tableParams, result);
@@ -160,19 +161,35 @@ public final class HiveTextProperties {
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
+ String escapeChar = getCsvCharacter(params, tableParams, ESCAPE_CHAR,
'"');
+ // Hive treats the writer-default double quote as a sentinel:
newReader selects the constructor
+ // whose effective escape is backslash. Resolve it before validating
the parser's character tuple.
+ if ("\"".equals(escapeChar)) {
+ escapeChar = "\\";
+ }
+ if ("\0".equals(separator)) {
+ throw new DorisConnectorException("Invalid OpenCSVSerde property
'separatorChar': must not be NUL");
+ }
+ // OpenCSV requires distinct active characters; NUL disables
quote/escape and may be shared by both.
+ if (separator.equals(quoteChar) || separator.equals(escapeChar)
+ || (!"\0".equals(quoteChar) && quoteChar.equals(escapeChar))) {
+ throw new DorisConnectorException("Invalid OpenCSVSerde
configuration: "
+ + "separatorChar, quoteChar and escapeChar must be
distinct when non-NUL");
+ }
+ result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR, separator);
+ // TextInputFormat owns physical records; neither table nor SerDe
line.delim changes its reader.
+ result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
DEFAULT_LINE_DELIM);
+ result.put(ScanNodePropertyKeys.TEXT_HIVE_OPEN_CSV, "true");
+ // Smooth-upgrade sources may still use the pre-flag reader at the new
query execution version.
+ result.put(ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS,
"Hive OpenCSVSerde semantics");
result.put(ScanNodePropertyKeys.TEXT_ENCLOSE, quoteChar);
- // #65501: BE strips the wrapping quotes only when the enclose char is
exactly the double-quote '"'.
- // The connector owns this CSV serde semantics, so decide here and
pass an explicit flag; the generic
- // PluginDrivenScanNode then just applies it instead of trimming for
any enclose char. Compare the
- // first byte, matching how the node sets enclose
(enclose.getBytes()[0]) and BE's getEnclose() == '"'.
- boolean trimDoubleQuotes = !quoteChar.isEmpty() &&
quoteChar.getBytes()[0] == (byte) '"';
- result.put(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES,
String.valueOf(trimDoubleQuotes));
- String escapeChar = getParamOrDefault(params, ESCAPE_CHAR, "\\");
+ // BE's extra double-quote trimming is valid only for the effective
double-quote enclosure.
+ result.put(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES,
String.valueOf("\"".equals(quoteChar)));
result.put(ScanNodePropertyKeys.TEXT_ESCAPE, escapeChar);
result.put(ScanNodePropertyKeys.TEXT_NULL_FORMAT, "");
}
@@ -203,10 +220,6 @@ public final class HiveTextProperties {
return supportMultiChar ? delim : getByte(delim, DEFAULT_FIELD_DELIM);
}
- private static String getLineDelimiter(Map<String, String> params) {
- return getParamOrDefault(params, LINE_DELIM, "\n");
- }
-
/**
* Looks up a SerDe property mirroring legacy {@code
HiveMetaStoreClientHelper.getSerdeProperty}:
* table parameters take precedence over StorageDescriptor/SerDeInfo
parameters, and the keys are
@@ -257,12 +270,28 @@ public final class HiveTextProperties {
}
}
- private static String getParamOrDefault(Map<String, String> params,
- String key, String defaultVal) {
- if (params == null) {
- return defaultVal;
+ private static String getCsvCharacter(Map<String, String> params,
Map<String, String> tableParams,
+ String key, char defaultValue) {
+ String value = serdeVal(params, tableParams, key);
+ if (value == null) {
+ return Character.toString(defaultValue);
+ }
+ if (value.isEmpty()) {
+ throw new DorisConnectorException("Invalid OpenCSVSerde property
'" + key + "': value must not be empty");
+ }
+ // Hive uses charAt(0), not a complete string, Unicode code point, or
numeric byte value.
+ // Validate only that effective character, after resolving
table-over-SerDe precedence.
+ char character = value.charAt(0);
+ if (Character.isSurrogate(character)) {
+ throw new DorisConnectorException("Unsupported OpenCSVSerde
property '" + key
+ + "': the first Java character must not be a surrogate");
+ }
+ // Separator is a Thrift string, but quote/escape are i8 fields
consumed as single UTF-8 bytes.
+ // Reject unrepresentable characters instead of silently passing only
their first encoded byte.
+ if (!SEPARATOR_CHAR.equals(key) && character > 0x7f) {
+ throw new DorisConnectorException("Unsupported OpenCSVSerde
property '" + key
+ + "': the first character must be ASCII (a single UTF-8
byte)");
}
- String val = params.get(key);
- return (val != null) ? val : defaultVal;
+ return Character.toString(character);
}
}
diff --git
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveCsvReaderCompatibilityTest.java
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveCsvReaderCompatibilityTest.java
new file mode 100644
index 00000000000..38c0557ef95
--- /dev/null
+++
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveCsvReaderCompatibilityTest.java
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.hive;
+
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ScanNodePropertyKeys;
+
+import au.com.bytecode.opencsv.CSVReader;
+import org.apache.hadoop.hive.serde2.OpenCSVSerde;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.io.Text;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.io.StringReader;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HexFormat;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+class HiveCsvReaderCompatibilityTest {
+ @Test
+ void testExplicitDefaultEscapeReadsLikeHive() throws Exception {
+ for (String quote : new String[] {"\"", "q"}) {
+ String record = quote + "left\\" + quote + "middle" + quote +
",tail";
+ Map<String, String> properties = Map.of("quoteChar", quote,
"escapeChar", "\"suffix");
+ List<?> expected = (List<?>) hiveSerde(properties).deserialize(new
Text(record));
+ Assertions.assertEquals(List.of("left" + quote + "middle",
"tail"), expected);
+ for (Map<String, String> extracted : List.of(
+
HiveTextProperties.extract(HiveTextProperties.HIVE_OPEN_CSV_SERDE, Map.of(),
properties),
+
HiveTextProperties.extract(HiveTextProperties.HIVE_OPEN_CSV_SERDE, properties,
Map.of()))) {
+ Assertions.assertEquals("\\",
extracted.get(ScanNodePropertyKeys.TEXT_ESCAPE));
+ try (CSVReader reader = new CSVReader(new StringReader(record),
+
extracted.get(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR).charAt(0),
+
extracted.get(ScanNodePropertyKeys.TEXT_ENCLOSE).charAt(0),
+
extracted.get(ScanNodePropertyKeys.TEXT_ESCAPE).charAt(0))) {
+ Assertions.assertEquals(expected,
Arrays.asList(reader.readNext()));
+ }
+ }
+ }
+ }
+
+ @Test
+ void testCharacterTuplesMatchHiveReaderAcceptance() throws Exception {
+ // Use Hive's real reader as the oracle, including its writer-default
escape sentinel and NUL rules.
+ char[] characters = {'\0', ',', '|', '"', '\\', 'q', 'e'};
+ for (char separator : characters) {
+ for (char quote : characters) {
+ for (char escape : characters) {
+ Map<String, String> properties = Map.of("separatorChar",
String.valueOf(separator),
+ "quoteChar", String.valueOf(quote), "escapeChar",
String.valueOf(escape));
+ OpenCSVSerde serde = hiveSerde(properties);
+ boolean accepted;
+ try {
+ serde.deserialize(new Text(""));
+ accepted = true;
+ } catch (SerDeException e) {
+
Assertions.assertInstanceOf(UnsupportedOperationException.class, e.getCause());
+ accepted = false;
+ }
+ assertAcceptance(accepted, Map.of(), properties);
+ assertAcceptance(accepted, properties, Map.of());
+ assertAcceptance(accepted, Map.of("quoteChar",
String.valueOf(quote)),
+ Map.of("separatorChar", String.valueOf(separator),
"escapeChar", String.valueOf(escape)));
+ }
+ }
+ }
+ }
+
+ @Test
+ void testRecordOracle() throws Exception {
+ // This corpus is also consumed by FileScannerV2. Generate
expectations with Hive itself,
+ // including null trailing fields and binary NULs, rather than a
second hand-written parser.
+ List<String> records = new ArrayList<>(List.of("x| qa|bq|c",
"qaqqbq,tail", "eeabc,tail",
+ "qleft", "rightq|tail", "abcqleft|rightq|tail", "a\0b,tail",
"", ",", "||",
+ "left|qunclosed", "qleftq|tail", "qleftqqrightq|tail", "x|\t
qa|bq|c",
+ "x|\u2003\u2003qa|bq|c", "x|\u00a0qa|bq|c",
"éqleft|rightq|tail",
+ "😀qleft|rightq|tail", "abcqleftérightqétail", "qéqétail",
"\"\"\"x\"\",tail"));
+ Random random = new Random(7321);
+ String alphabet = "aqe|, \\" + '"' + '\0';
+ for (int row = 0; row < 80; row++) {
+ StringBuilder record = new StringBuilder();
+ for (int i = 0, length = random.nextInt(24); i < length; i++) {
+
record.append(alphabet.charAt(random.nextInt(alphabet.length())));
+ }
+ records.add(record.toString());
+ }
+ List<String> corpus = new ArrayList<>();
+ for (String tuple : List.of("|qe", ",q\0", ",\0e", ",\0\0", ",\"\\",
"éqe")) {
+ OpenCSVSerde serde = hiveSerde(Map.of("separatorChar",
tuple.substring(0, 1),
+ "quoteChar", tuple.substring(1, 2), "escapeChar",
tuple.substring(2, 3)), 3);
+ for (String record : records) {
+ List<?> result = (List<?>) serde.deserialize(new Text(record));
+ StringBuilder encoded = new
StringBuilder(hex(tuple)).append('\t').append(hex(record));
+ for (Object value : result) {
+ encoded.append('\t').append(value == null ? "NULL" :
hex(value.toString()));
+ }
+ corpus.add(encoded.toString());
+ }
+ }
+ String output = System.getProperty("hive.csv.oracle.output");
+ if (output != null) {
+ Files.write(Path.of(output), corpus, StandardCharsets.UTF_8);
+ }
+ try (var input =
getClass().getResourceAsStream("/hive-csv-oracle.tsv")) {
+ Assertions.assertNotNull(input);
+ Assertions.assertEquals(corpus,
+ new String(input.readAllBytes(),
StandardCharsets.UTF_8).lines().toList());
+ }
+ }
+
+ private static String hex(String value) {
+ return value.isEmpty() ? "EMPTY" :
HexFormat.of().formatHex(value.getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static void assertAcceptance(boolean accepted, Map<String, String>
serdeProperties,
+ Map<String, String> tableProperties) {
+ if (accepted) {
+ Assertions.assertDoesNotThrow(() -> HiveTextProperties.extract(
+ HiveTextProperties.HIVE_OPEN_CSV_SERDE, serdeProperties,
tableProperties));
+ } else {
+ Assertions.assertThrows(DorisConnectorException.class, () ->
HiveTextProperties.extract(
+ HiveTextProperties.HIVE_OPEN_CSV_SERDE, serdeProperties,
tableProperties));
+ }
+ }
+
+ private static OpenCSVSerde hiveSerde(Map<String, String> properties)
throws SerDeException {
+ return hiveSerde(properties, 2);
+ }
+
+ private static OpenCSVSerde hiveSerde(Map<String, String> properties, int
columns) throws SerDeException {
+ Properties serdeProperties = new Properties();
+ serdeProperties.setProperty("columns", columns == 2 ? "first,second" :
"first,second,third");
+ serdeProperties.setProperty("columns.types", columns == 2 ?
"string:string" : "string:string:string");
+ serdeProperties.putAll(properties);
+ OpenCSVSerde serde = new OpenCSVSerde();
+ serde.initialize(null, serdeProperties);
+ return serde;
+ }
+}
diff --git
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java
index 720d9b91066..2c1937e9739 100644
---
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java
+++
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java
@@ -31,6 +31,7 @@ import
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider;
import org.apache.doris.connector.spi.scan.ConnectorScanProfile;
import org.apache.doris.connector.spi.scan.ConnectorScanRange;
import org.apache.doris.connector.spi.scan.ConnectorScanRequest;
+import org.apache.doris.connector.spi.scan.ScanNodePropertyKeys;
import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.thrift.TFileCompressType;
import org.apache.doris.thrift.TFileScanRangeParams;
@@ -418,6 +419,30 @@ public class HiveScanBatchModeTest {
Assertions.assertEquals("aliasAK",
props.get("location.s3.access_key"));
}
+ @Test
+ public void
getScanNodePropertiesUsesCsvTableParametersForPartitionedAndUnpartitionedTables()
{
+ HiveScanPlanProvider provider = provider(new FakeHmsClient(), new
CountingLister());
+ for (List<String> partitionKeys :
Arrays.asList(Collections.<String>emptyList(),
+ Collections.singletonList("bucket"))) {
+ HiveTableHandle handle = new HiveTableHandle.Builder("db",
"csv_table", HiveTableType.HIVE)
+ .inputFormat("org.apache.hadoop.mapred.TextInputFormat")
+ .serializationLib(HiveTextProperties.HIVE_OPEN_CSV_SERDE)
+ .partitionKeyNames(partitionKeys)
+ .tableParameters(Map.of("separatorChar", "s", "quoteChar",
"q", "escapeChar", "e"))
+ .build();
+ Map<String, String> props = provider.getScanNodeProperties(
+ new FakeSession(), handle, Collections.emptyList(),
Optional.empty());
+ Assertions.assertAll(
+ () -> Assertions.assertEquals("csv",
props.get(ScanNodePropertyKeys.FILE_FORMAT_TYPE)),
+ () -> Assertions.assertEquals("s",
props.get(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR)),
+ () -> Assertions.assertEquals("q",
props.get(ScanNodePropertyKeys.TEXT_ENCLOSE)),
+ () -> Assertions.assertEquals("e",
props.get(ScanNodePropertyKeys.TEXT_ESCAPE)),
+ () -> Assertions.assertEquals("false",
props.get(ScanNodePropertyKeys.TEXT_TRIM_DOUBLE_QUOTES)));
+ Assertions.assertEquals(partitionKeys.isEmpty() ? null : "bucket",
+ props.get(ScanNodePropertyKeys.PATH_PARTITION_KEYS));
+ }
+ }
+
// ============ #65437: scan-level transactional_hive marker
(FileScannerV2 exclusion) ============
// Sibling to supportsBatchScan's ACID exclusion above. BE picks
FileScannerV2 from SCAN-LEVEL params before the
// per-range splits arrive, and V2 does not apply ACID delete deltas. A
full-ACID hive scan must therefore carry
diff --git
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java
index 8723ad4d0a4..702ae5c287d 100644
---
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java
+++
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java
@@ -17,6 +17,8 @@
package org.apache.doris.connector.hive;
+import org.apache.doris.connector.spi.DorisConnectorException;
+
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -176,4 +178,171 @@ public class HiveTextPropertiesTest {
Map<String, String> r = HiveTextProperties.extract(OPEN_CSV_SERDE,
sd("quoteChar", "\""), new HashMap<>());
Assertions.assertEquals("true", r.get(PREFIX + "trim_double_quotes"));
}
+
+ @Test
+ public void testCsvTableParameters() {
+ Map<String, String> result =
HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
+ sd("separatorChar", "s", "quoteChar", "q", "escapeChar", "e"));
+ assertCsvProperties(result, "s", "q", "e", "false");
+ }
+
+ @Test
+ public void testCsvTableParametersOverrideSerdeParameters() {
+ Map<String, String> result = HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("separatorChar", ",", "quoteChar", "\"", "escapeChar",
"\\"),
+ sd("separatorChar", "s", "quoteChar", "q", "escapeChar", "e"));
+ assertCsvProperties(result, "s", "q", "e", "false");
+ }
+
+ @Test
+ public void testCsvPropertiesFallBackIndividually() {
+ Map<String, String> result = HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("separatorChar", "s", "quoteChar", "q"), sd("quoteChar",
"\""));
+ assertCsvProperties(result, "s", "\"", "\\", "true");
+ }
+
+ @Test
+ public void testCsvSerdeParameters() {
+ Map<String, String> result = HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("separatorChar", "s", "quoteChar", "q", "escapeChar", "e"),
null);
+ assertCsvProperties(result, "s", "q", "e", "false");
+ }
+
+ @Test
+ public void testCsvDefaultsWithMissingParameterMaps() {
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, null,
null),
+ ",", "\"", "\\", "true");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, null,
+ sd("separatorChar", "s", "quoteChar", "q", "escapeChar", "e")),
+ "s", "q", "e", "false");
+ }
+
+ @Test
+ public void testCsvIgnoresTableAndSerdeRecordDelimiters() {
+ for (String value : new String[] {"|", "", "\r\n"}) {
+ Map<String, String> result =
HiveTextProperties.extract(OPEN_CSV_SERDE, sd(), sd("line.delim", value));
+ Assertions.assertEquals("\n", result.get(PREFIX +
"line_delimiter"));
+ }
+ Map<String, String> result = HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("line.delim", "\r\n"), sd("line.delim", "|"));
+ Assertions.assertEquals("\n", result.get(PREFIX + "line_delimiter"));
+ }
+
+ @Test
+ public void testCsvUsesFirstJavaCharacterFromEitherPropertySource() {
+ Map<String, String> properties = sd("separatorChar", "ss",
"quoteChar", "qq", "escapeChar", "ee");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
properties),
+ "s", "q", "e", "false");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
properties, sd()),
+ "s", "q", "e", "false");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
+ sd("separatorChar", "99", "quoteChar", "\"suffix",
"escapeChar", "eé")),
+ "9", "\"", "e", "true");
+ }
+
+ @Test
+ public void testCsvRejectsEmptyCharacterPropertiesInsteadOfFallingBack() {
+ for (String key : new String[] {"separatorChar", "quoteChar",
"escapeChar"}) {
+ DorisConnectorException tableError =
Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE, sd(key,
"x"), sd(key, "")));
+ Assertions.assertTrue(tableError.getMessage().contains(key));
+ Assertions.assertTrue(tableError.getMessage().contains("empty"));
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE, sd(key,
""), sd()));
+ }
+ }
+
+ @Test
+ public void testCsvRejectsNonAsciiQuoteAndEscapeFromEitherSource() {
+ for (String key : new String[] {"quoteChar", "escapeChar"}) {
+ for (String value : new String[] {"é", "中", "😀"}) {
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
sd(key, value)), key);
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE,
sd(key, value), sd()), key);
+ }
+ }
+ }
+
+ @Test
+ public void testCsvPreservesUtf8SeparatorButRejectsSurrogates() {
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
sd("separatorChar", "ésuffix")),
+ "é", "\"", "\\", "true");
+ for (String value : new String[] {"😀",
String.valueOf(Character.MIN_HIGH_SURROGATE),
+ String.valueOf(Character.MIN_LOW_SURROGATE)}) {
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
sd("separatorChar", value)));
+ }
+ }
+
+ @Test
+ public void testCsvValidatesOnlyTheEffectiveCharacterProperties() {
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("separatorChar", "", "quoteChar", "é", "escapeChar", ""),
+ sd("separatorChar", "ss", "quoteChar", "qq", "escapeChar",
"ee")),
+ "s", "q", "e", "false");
+ }
+
+ @Test
+ public void testCsvDefaultEscapeSentinelFromEitherPropertySource() {
+ for (String value : new String[] {"\"", "\"suffix"}) {
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
sd(), sd("escapeChar", value)),
+ ",", "\"", "\\", "true");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
sd("escapeChar", value), sd()),
+ ",", "\"", "\\", "true");
+ }
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("escapeChar", "e"), sd("escapeChar", "\"", "quoteChar",
"q")),
+ ",", "q", "\\", "false");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("escapeChar", "\""), sd("escapeChar", "e")), ",", "\"",
"e", "true");
+ }
+
+ @Test
+ public void testCsvRejectsConflictingEffectiveCharacters() {
+ String[][] pairs = {{"separatorChar", "quoteChar"}, {"separatorChar",
"escapeChar"},
+ {"quoteChar", "escapeChar"}};
+ for (String[] pair : pairs) {
+ Map<String, String> properties = sd(pair[0], "|first", pair[1],
"|second");
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
properties));
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE,
properties, sd()));
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE,
sd(pair[0], "|"), sd(pair[1], "|")));
+ }
+ // Resolve the writer-default escape sentinel before checking
parser-character conflicts.
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("separatorChar", "\\"), sd("escapeChar", "\"")));
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("quoteChar", "\\"), sd("escapeChar", "\"")));
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
+ sd("separatorChar", "\"", "quoteChar", "q", "escapeChar",
"\"")),
+ "\"", "q", "\\", "false");
+ }
+
+ @Test
+ public void testCsvRejectsNulSeparatorButAllowsDisabledQuoteAndEscape() {
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
sd("separatorChar", "\0")));
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> HiveTextProperties.extract(OPEN_CSV_SERDE,
sd("separatorChar", "\0"), sd()));
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE, sd(),
+ sd("quoteChar", "\0", "escapeChar", "\0")), ",", "\0", "\0",
"false");
+ assertCsvProperties(HiveTextProperties.extract(OPEN_CSV_SERDE,
+ sd("quoteChar", "\0", "escapeChar", "\0"), sd()), ",", "\0",
"\0", "false");
+ }
+
+ private static void assertCsvProperties(Map<String, String> result,
+ String separator, String quote, String escape, String
trimDoubleQuotes) {
+ Assertions.assertAll(
+ () -> Assertions.assertEquals(separator, result.get(PREFIX +
"column_separator")),
+ () -> Assertions.assertEquals(quote, result.get(PREFIX +
"enclose")),
+ () -> Assertions.assertEquals(escape, result.get(PREFIX +
"escape")),
+ () -> Assertions.assertEquals(trimDoubleQuotes,
result.get(PREFIX + "trim_double_quotes")),
+ () -> Assertions.assertEquals("\n", result.get(PREFIX +
"line_delimiter")),
+ () -> Assertions.assertEquals("", result.get(PREFIX +
"null_format")));
+ }
}
diff --git
a/fe/fe-connector/fe-connector-hive/src/test/resources/hive-csv-oracle.tsv
b/fe/fe-connector/fe-connector-hive/src/test/resources/hive-csv-oracle.tsv
new file mode 100644
index 00000000000..c07372f3d64
--- /dev/null
+++ b/fe/fe-connector/fe-connector-hive/src/test/resources/hive-csv-oracle.tsv
@@ -0,0 +1,606 @@
+7c7165 787c202071617c62717c63 78 617c62 63
+7c7165 7161717162712c7461696c 617162712c7461696c NULL NULL
+7c7165 65656162632c7461696c 6162632c7461696c NULL NULL
+7c7165 716c656674 NULL NULL NULL
+7c7165 7269676874717c7461696c NULL NULL NULL
+7c7165 616263716c6566747c7269676874717c7461696c
616263716c66747c7269676874 7461696c NULL
+7c7165 6100622c7461696c 6100622c7461696c NULL NULL
+7c7165 EMPTY NULL NULL NULL
+7c7165 2c 2c NULL NULL
+7c7165 7c7c EMPTY EMPTY EMPTY
+7c7165 6c6566747c71756e636c6f736564 6c6674 NULL NULL
+7c7165 716c656674717c7461696c 6c6674 7461696c NULL
+7c7165 716c65667471717269676874717c7461696c 6c6674717269676874
7461696c NULL
+7c7165 787c092071617c62717c63 78 617c62 63
+7c7165 787ce28083e2808371617c62717c63 78 617c62 63
+7c7165 787cc2a071617c62717c63 78 c2a071617c62 63
+7c7165 c3a9716c6566747c7269676874717c7461696c c3a96c66747c7269676874
7461696c NULL
+7c7165 f09f9880716c6566747c7269676874717c7461696c
f09f98806c66747c7269676874 7461696c NULL
+7c7165 616263716c656674c3a9726967687471c3a97461696c
616263716c6674c3a9726967687471c3a97461696c NULL NULL
+7c7165 71c3a971c3a97461696c c3a9c3a97461696c NULL NULL
+7c7165 2222227822222c7461696c 2222227822222c7461696c NULL NULL
+7c7165 6520007c205c2000616120202c5c2c 2000 205c2000616120202c5c2c NULL
+7c7165 20 20 NULL NULL
+7c7165 EMPTY NULL NULL NULL
+7c7165 222061205c7c20717c65005c205c71715c00617c71 222061205c
207c005c205c715c00617c NULL
+7c7165 007161202c7c7161612071652c2071 0061202c7c616120712c20 NULL NULL
+7c7165 71007c0065657c00616100207c7c NULL NULL NULL
+7c7165 22710022617161612020 220022617161612020 NULL NULL
+7c7165 5c7c6161617c20716520205c2c 5c 616161 NULL
+7c7165 65 EMPTY NULL NULL
+7c7165 00652c227c6561222200222c2222207165617c2265005c 002c22 NULL NULL
+7c7165 5c7165225c71226161 5c225c71226161 NULL NULL
+7c7165 7c005c2c007c22612c2c65 EMPTY 005c2c00 22612c2c
+7c7165 612c5c20205c7c5c7c00226520717c2c655c655c2c6122 612c5c20205c 5c
NULL
+7c7165 007c6161200022615c202061227122 00 NULL NULL
+7c7165 2c5c20002c5c5c2c225c5c71 NULL NULL NULL
+7c7165 2265652022712c712c00657120202261207c5c 22652022712c712c00712020226120
5c NULL
+7c7165 657c5c225c0065225c65656565 EMPTY 5c225c00225c6565 NULL
+7c7165 2c5c715c5c20225c71227c5c00656120717c7c5c61 2c5c5c5c20225c7122
NULL NULL
+7c7165 206561612c20000065652265205c00 2061612c2000006522205c00 NULL
NULL
+7c7165 00002c2c5c20615c5c00 00002c2c5c20615c5c00 NULL NULL
+7c7165 717c6161222c NULL NULL NULL
+7c7165 0020226171655c5c NULL NULL NULL
+7c7165 5c007c22 5c00 22 NULL
+7c7165 0022202261 0022202261 NULL NULL
+7c7165 6500617c220020 0061 220020 NULL
+7c7165 20655c7c5c 205c 5c NULL
+7c7165 20612c2c71006171 20612c2c710061 NULL NULL
+7c7165 202222717c7c5c7120202222226565617c205c 2022227c7c5c7120202222226561
205c NULL
+7c7165 205c2c007120207c5c005c20002c22007c5c5c5c22 NULL NULL NULL
+7c7165 7c20712200652c7c2c2c EMPTY NULL NULL
+7c7165 65202c005c2061007161227c5c7c2c717c65657c61
202c005c2061007161227c5c7c2c EMPTY 61
+7c7165 00227c7c7c715c652c0020712c71615c 0022 EMPTY EMPTY
+7c7165 007c 00 EMPTY NULL
+7c7165 2c0020206120617171716561612c5c20 NULL NULL NULL
+7c7165 657161205c22 NULL NULL NULL
+7c7165 655c202c65207c7165612271225c657c 5c202c20 612271225c
EMPTY
+7c7165 6520612022612c2261 20612022612c2261 NULL NULL
+7c7165 2c2c71007c71615c7c22715c2271 2c2c007c615c 22715c22 NULL
+7c7165 EMPTY NULL NULL NULL
+7c7165 6561225c 61225c NULL NULL
+7c7165 7c617c225c22206520 EMPTY 61 225c222020
+7c7165 EMPTY NULL NULL NULL
+7c7165 61 61 NULL NULL
+7c7165 227c00 22 00 NULL
+7c7165 61612c00615c2c5c5c226520202c002061007c5c71
61612c00615c2c5c5c2220202c00206100 NULL NULL
+7c7165 2000 2000 NULL NULL
+7c7165 22005c7c5c6171657c71656120006161 22005c 5c61717c6120006161
NULL
+7c7165 617c2c 61 2c NULL
+7c7165 715c612c2c61 NULL NULL NULL
+7c7165 2c207c2c225c2c7161 2c20 NULL NULL
+7c7165 657c652c2c6500002200656565207c6520 EMPTY 2c2c000022006520
20
+7c7165 7122005c22207171612020712c2c61615c206122712c65 NULL NULL NULL
+7c7165 2c715c71657c61716122657c 2c5c71 NULL NULL
+7c7165 00 00 NULL NULL
+7c7165 2061 2061 NULL NULL
+7c7165 205c2000 205c2000 NULL NULL
+7c7165 71655c2c207c20715c0000006500227c6161 5c2c207c20715c0000000022
6161 NULL
+7c7165 5c6571 5c71 NULL NULL
+7c7165 007c2c652c7171 00 2c2c71 NULL
+7c7165 00615c 00615c NULL NULL
+7c7165 71656522006565225c00 NULL NULL NULL
+7c7165 002c7165205c65227c6165716565207c716171612c2c NULL NULL NULL
+7c7165 222c71206120005c612c2c NULL NULL NULL
+7c7165 7c227c617c7c22206500712022002c7c00652c5c EMPTY 22 61
+7c7165 61612c225c5c20652222205c2071222261 NULL NULL NULL
+7c7165 61 61 NULL NULL
+7c7165 205c2c5c2c 205c2c5c2c NULL NULL
+7c7165 2c6500007c22 2c0000 22 NULL
+7c7165 5c7c2c65202c5c7c610022227c225c 5c 2c202c5c 61002222
+7c7165 655c2c002c7165716561205c7100207c2c7171225c2c20
5c2c002c717161205c710020 2c71225c2c20 NULL
+7c7165 2c22 2c22 NULL NULL
+7c7165 6571 NULL NULL NULL
+7c7165 5c205c 5c205c NULL NULL
+7c7165 65715c0071615c7c2271207c5c7161207c00207c65 5c0071615c
2271207c5c716120 0020
+7c7165 205c20615c220000222c615c5c2c00225c71 NULL NULL NULL
+7c7165 2c7c5c20657c5c7c202c 2c 5c20 5c
+7c7165 5c65002c005c5c2c2220 5c002c005c5c2c2220 NULL NULL
+7c7165 71 NULL NULL NULL
+7c7165 71617c7c65615c61715c2c652200 617c7c615c61715c2c2200 NULL NULL
+7c7165 7c7165715c5c5c612065655c EMPTY NULL NULL
+2c7100 787c202071617c62717c63 787c202071617c62717c63 NULL NULL
+2c7100 7161717162712c7461696c 617162 7461696c NULL
+2c7100 65656162632c7461696c 6565616263 7461696c NULL
+2c7100 716c656674 NULL NULL NULL
+2c7100 7269676874717c7461696c NULL NULL NULL
+2c7100 616263716c6566747c7269676874717c7461696c
616263716c6566747c7269676874717c7461696c NULL NULL
+2c7100 6100622c7461696c 6162 7461696c NULL
+2c7100 EMPTY NULL NULL NULL
+2c7100 2c EMPTY EMPTY NULL
+2c7100 7c7c 7c7c NULL NULL
+2c7100 6c6566747c71756e636c6f736564 NULL NULL NULL
+2c7100 716c656674717c7461696c 6c656674717c7461696c NULL NULL
+2c7100 716c65667471717269676874717c7461696c
6c656674717269676874717c7461696c NULL NULL
+2c7100 787c092071617c62717c63 787c092071617c62717c63 NULL NULL
+2c7100 787ce28083e2808371617c62717c63 787ce28083e2808371617c62717c63 NULL
NULL
+2c7100 787cc2a071617c62717c63 787cc2a071617c62717c63 NULL NULL
+2c7100 c3a9716c6566747c7269676874717c7461696c
c3a96c6566747c7269676874717c7461696c NULL NULL
+2c7100 f09f9880716c6566747c7269676874717c7461696c
f09f98806c6566747c7269676874717c7461696c NULL NULL
+2c7100 616263716c656674c3a9726967687471c3a97461696c
616263716c656674c3a9726967687471c3a97461696c NULL NULL
+2c7100 71c3a971c3a97461696c c3a9c3a97461696c NULL NULL
+2c7100 2222227822222c7461696c 222222782222 7461696c NULL
+2c7100 6520007c205c2000616120202c5c2c 65207c205c2061612020 5c EMPTY
+2c7100 20 20 NULL NULL
+2c7100 EMPTY NULL NULL NULL
+2c7100 222061205c7c20717c65005c205c71715c00617c71
222061205c7c20717c655c205c715c617c NULL NULL
+2c7100 007161202c7c7161612071652c2071 61202c7c7161612071652c20 NULL
NULL
+2c7100 71007c0065657c00616100207c7c NULL NULL NULL
+2c7100 22710022617161612020 2222617161612020 NULL NULL
+2c7100 5c7c6161617c20716520205c2c NULL NULL NULL
+2c7100 65 65 NULL NULL
+2c7100 00652c227c6561222200222c2222207165617c2265005c 65 227c6561222222
NULL
+2c7100 5c7165225c71226161 5c65225c71226161 NULL NULL
+2c7100 7c005c2c007c22612c2c65 7c5c 7c2261 EMPTY
+2c7100 612c5c20205c7c5c7c00226520717c2c655c655c2c6122 61 NULL NULL
+2c7100 007c6161200022615c202061227122 NULL NULL NULL
+2c7100 2c5c20002c5c5c2c225c5c71 EMPTY 5c20 5c5c
+2c7100 2265652022712c712c00657120202261207c5c 22656520222c NULL NULL
+2c7100 657c5c225c0065225c65656565 657c5c225c65225c65656565 NULL
NULL
+2c7100 2c5c715c5c20225c71227c5c00656120717c7c5c61 EMPTY NULL NULL
+2c7100 206561612c20000065652265205c00 20656161 200065652265205c
NULL
+2c7100 00002c2c5c20615c5c00 EMPTY EMPTY 5c20615c5c
+2c7100 717c6161222c NULL NULL NULL
+2c7100 0020226171655c5c NULL NULL NULL
+2c7100 5c007c22 5c7c22 NULL NULL
+2c7100 0022202261 22202261 NULL NULL
+2c7100 6500617c220020 65617c2220 NULL NULL
+2c7100 20655c7c5c 20655c7c5c NULL NULL
+2c7100 20612c2c71006171 2061 EMPTY 61
+2c7100 202222717c7c5c7120202222226565617c205c
202222717c7c5c7120202222226565617c205c NULL NULL
+2c7100 205c2c007120207c5c005c20002c22007c5c5c5c22 205c NULL NULL
+2c7100 7c20712200652c7c2c2c NULL NULL NULL
+2c7100 65202c005c2061007161227c5c7c2c717c65657c61 6520
5c20617161227c5c7c NULL
+2c7100 00227c7c7c715c652c0020712c71615c 227c7c7c715c652c20 NULL
NULL
+2c7100 007c 7c NULL NULL
+2c7100 2c0020206120617171716561612c5c20 EMPTY NULL NULL
+2c7100 657161205c22 NULL NULL NULL
+2c7100 655c202c65207c7165612271225c657c 655c20
65207c7165612271225c657c NULL
+2c7100 6520612022612c2261 652061202261 2261 NULL
+2c7100 2c2c71007c71615c7c22715c2271 EMPTY EMPTY 7c71615c7c22715c22
+2c7100 EMPTY NULL NULL NULL
+2c7100 6561225c 6561225c NULL NULL
+2c7100 7c617c225c22206520 7c617c225c22206520 NULL NULL
+2c7100 EMPTY NULL NULL NULL
+2c7100 61 61 NULL NULL
+2c7100 227c00 227c NULL NULL
+2c7100 61612c00615c2c5c5c226520202c002061007c5c71 6161 615c
5c5c22652020
+2c7100 2000 20 NULL NULL
+2c7100 22005c7c5c6171657c71656120006161 225c7c5c6171657c716561206161
NULL NULL
+2c7100 617c2c 617c EMPTY NULL
+2c7100 715c612c2c61 NULL NULL NULL
+2c7100 2c207c2c225c2c7161 EMPTY 207c 225c
+2c7100 657c652c2c6500002200656565207c6520 657c65 EMPTY
650022656565207c6520
+2c7100 7122005c22207171612020712c2c61615c206122712c65 225c222071612020
EMPTY NULL
+2c7100 2c715c71657c61716122657c EMPTY NULL NULL
+2c7100 00 EMPTY NULL NULL
+2c7100 2061 2061 NULL NULL
+2c7100 205c2000 205c20 NULL NULL
+2c7100 71655c2c207c20715c0000006500227c6161 655c2c207c20715c0065227c6161
NULL NULL
+2c7100 5c6571 NULL NULL NULL
+2c7100 007c2c652c7171 7c 65 EMPTY
+2c7100 00615c 615c NULL NULL
+2c7100 71656522006565225c00 NULL NULL NULL
+2c7100 002c7165205c65227c6165716565207c716171612c2c EMPTY
65205c65227c6165716565207c71617161 EMPTY
+2c7100 222c71206120005c612c2c 22 NULL NULL
+2c7100 7c227c617c7c22206500712022002c7c00652c5c
7c227c617c7c222065712022 7c65 5c
+2c7100 61612c225c5c20652222205c2071222261 6161 NULL NULL
+2c7100 61 61 NULL NULL
+2c7100 205c2c5c2c 205c 5c EMPTY
+2c7100 2c6500007c22 EMPTY 65007c22 NULL
+2c7100 5c7c2c65202c5c7c610022227c225c 5c7c 6520 5c7c6122227c225c
+2c7100 655c2c002c7165716561205c7100207c2c7171225c2c20 655c EMPTY NULL
+2c7100 2c22 EMPTY 22 NULL
+2c7100 6571 NULL NULL NULL
+2c7100 5c205c 5c205c NULL NULL
+2c7100 65715c0071615c7c2271207c5c7161207c00207c65 NULL NULL NULL
+2c7100 205c20615c220000222c615c5c2c00225c71 205c20615c220022 615c5c
NULL
+2c7100 2c7c5c20657c5c7c202c EMPTY 7c5c20657c5c7c20 EMPTY
+2c7100 5c65002c005c5c2c2220 5c65 5c5c 2220
+2c7100 71 NULL NULL NULL
+2c7100 71617c7c65615c61715c2c652200 617c7c65615c61715c 6522 NULL
+2c7100 7c7165715c5c5c612065655c 7c65715c5c5c612065655c NULL NULL
+2c0065 787c202071617c62717c63 787c202071617c62717c63 NULL NULL
+2c0065 7161717162712c7461696c 716171716271 7461696c NULL
+2c0065 65656162632c7461696c 616263 7461696c NULL
+2c0065 716c656674 716c6674 NULL NULL
+2c0065 7269676874717c7461696c 7269676874717c7461696c NULL NULL
+2c0065 616263716c6566747c7269676874717c7461696c
616263716c66747c7269676874717c7461696c NULL NULL
+2c0065 6100622c7461696c NULL NULL NULL
+2c0065 EMPTY NULL NULL NULL
+2c0065 2c EMPTY EMPTY NULL
+2c0065 7c7c 7c7c NULL NULL
+2c0065 6c6566747c71756e636c6f736564 6c66747c71756e636c6f7364 NULL
NULL
+2c0065 716c656674717c7461696c 716c6674717c7461696c NULL NULL
+2c0065 716c65667471717269676874717c7461696c
716c667471717269676874717c7461696c NULL NULL
+2c0065 787c092071617c62717c63 787c092071617c62717c63 NULL NULL
+2c0065 787ce28083e2808371617c62717c63 787ce28083e2808371617c62717c63 NULL
NULL
+2c0065 787cc2a071617c62717c63 787cc2a071617c62717c63 NULL NULL
+2c0065 c3a9716c6566747c7269676874717c7461696c
c3a9716c66747c7269676874717c7461696c NULL NULL
+2c0065 f09f9880716c6566747c7269676874717c7461696c
f09f9880716c66747c7269676874717c7461696c NULL NULL
+2c0065 616263716c656674c3a9726967687471c3a97461696c
616263716c6674c3a9726967687471c3a97461696c NULL NULL
+2c0065 71c3a971c3a97461696c 71c3a971c3a97461696c NULL NULL
+2c0065 2222227822222c7461696c 222222782222 7461696c NULL
+2c0065 6520007c205c2000616120202c5c2c 207c205c200061612020 5c EMPTY
+2c0065 20 20 NULL NULL
+2c0065 EMPTY NULL NULL NULL
+2c0065 222061205c7c20717c65005c205c71715c00617c71 NULL NULL NULL
+2c0065 007161202c7c7161612071652c2071 NULL NULL NULL
+2c0065 71007c0065657c00616100207c7c 717c007c00616100207c7c NULL NULL
+2c0065 22710022617161612020 NULL NULL NULL
+2c0065 5c7c6161617c20716520205c2c 5c7c6161617c207120205c EMPTY NULL
+2c0065 65 EMPTY NULL NULL
+2c0065 00652c227c6561222200222c2222207165617c2265005c 2c227c6122220022
22222071617c22005c NULL
+2c0065 5c7165225c71226161 5c71225c71226161 NULL NULL
+2c0065 7c005c2c007c22612c2c65 7c5c2c7c2261 EMPTY EMPTY
+2c0065 612c5c20205c7c5c7c00226520717c2c655c655c2c6122 61 NULL NULL
+2c0065 007c6161200022615c202061227122 7c6161200022615c202061227122 NULL
NULL
+2c0065 2c5c20002c5c5c2c225c5c71 EMPTY NULL NULL
+2c0065 2265652022712c712c00657120202261207c5c 2265202271 71 NULL
+2c0065 657c5c225c0065225c65656565 NULL NULL NULL
+2c0065 2c5c715c5c20225c71227c5c00656120717c7c5c61 EMPTY NULL NULL
+2c0065 206561612c20000065652265205c00 206161 NULL NULL
+2c0065 00002c2c5c20615c5c00 EMPTY EMPTY NULL
+2c0065 717c6161222c 717c616122 EMPTY NULL
+2c0065 0020226171655c5c NULL NULL NULL
+2c0065 5c007c22 NULL NULL NULL
+2c0065 0022202261 NULL NULL NULL
+2c0065 6500617c220020 617c220020 NULL NULL
+2c0065 20655c7c5c 205c7c5c NULL NULL
+2c0065 20612c2c71006171 2061 EMPTY NULL
+2c0065 202222717c7c5c7120202222226565617c205c
202222717c7c5c71202022222265617c205c NULL NULL
+2c0065 205c2c007120207c5c005c20002c22007c5c5c5c22 205c
7120207c5c005c202c22007c5c5c5c22 NULL
+2c0065 7c20712200652c7c2c2c NULL NULL NULL
+2c0065 65202c005c2061007161227c5c7c2c717c65657c61 20
5c2061007161227c5c7c 717c657c61
+2c0065 00227c7c7c715c652c0020712c71615c 227c7c7c715c2c2071 71615c
NULL
+2c0065 007c NULL NULL NULL
+2c0065 2c0020206120617171716561612c5c20 EMPTY NULL NULL
+2c0065 657161205c22 7161205c22 NULL NULL
+2c0065 655c202c65207c7165612271225c657c 5c20 207c71612271225c7c
NULL
+2c0065 6520612022612c2261 2061202261 2261 NULL
+2c0065 2c2c71007c71615c7c22715c2271 EMPTY EMPTY NULL
+2c0065 EMPTY NULL NULL NULL
+2c0065 6561225c 61225c NULL NULL
+2c0065 7c617c225c22206520 7c617c225c222020 NULL NULL
+2c0065 EMPTY NULL NULL NULL
+2c0065 61 61 NULL NULL
+2c0065 227c00 NULL NULL NULL
+2c0065 61612c00615c2c5c5c226520202c002061007c5c71 6161 NULL NULL
+2c0065 2000 NULL NULL NULL
+2c0065 22005c7c5c6171657c71656120006161 225c7c5c61717c716120006161
NULL NULL
+2c0065 617c2c 617c EMPTY NULL
+2c0065 715c612c2c61 715c61 EMPTY 61
+2c0065 2c207c2c225c2c7161 EMPTY 207c 225c
+2c0065 657c652c2c6500002200656565207c6520 7c EMPTY NULL
+2c0065 7122005c22207171612020712c2c61615c206122712c65 NULL NULL NULL
+2c0065 2c715c71657c61716122657c EMPTY 715c717c617161227c NULL
+2c0065 00 NULL NULL NULL
+2c0065 2061 2061 NULL NULL
+2c0065 205c2000 NULL NULL NULL
+2c0065 71655c2c207c20715c0000006500227c6161 715c NULL NULL
+2c0065 5c6571 5c71 NULL NULL
+2c0065 007c2c652c7171 NULL NULL NULL
+2c0065 00615c NULL NULL NULL
+2c0065 71656522006565225c00 7165220065225c NULL NULL
+2c0065 002c7165205c65227c6165716565207c716171612c2c NULL NULL NULL
+2c0065 222c71206120005c612c2c 22 NULL NULL
+2c0065 7c227c617c7c22206500712022002c7c00652c5c
7c227c617c7c2220007120222c7c00 5c NULL
+2c0065 61612c225c5c20652222205c2071222261 6161
225c5c202222205c2071222261 NULL
+2c0065 61 61 NULL NULL
+2c0065 205c2c5c2c 205c 5c EMPTY
+2c0065 2c6500007c22 EMPTY 007c22 NULL
+2c0065 5c7c2c65202c5c7c610022227c225c 5c7c 20 NULL
+2c0065 655c2c002c7165716561205c7100207c2c7171225c2c20 5c
2c717161205c7100207c 7171225c
+2c0065 2c22 EMPTY 22 NULL
+2c0065 6571 71 NULL NULL
+2c0065 5c205c 5c205c NULL NULL
+2c0065 65715c0071615c7c2271207c5c7161207c00207c65
715c0071615c7c2271207c5c7161207c00207c NULL NULL
+2c0065 205c20615c220000222c615c5c2c00225c71 205c20615c220022 615c5c
NULL
+2c0065 2c7c5c20657c5c7c202c EMPTY 7c5c207c5c7c20 EMPTY
+2c0065 5c65002c005c5c2c2220 5c00 NULL NULL
+2c0065 71 71 NULL NULL
+2c0065 71617c7c65615c61715c2c652200 71617c7c615c61715c NULL NULL
+2c0065 7c7165715c5c5c612065655c 7c71715c5c5c6120655c NULL NULL
+2c0000 787c202071617c62717c63 787c202071617c62717c63 NULL NULL
+2c0000 7161717162712c7461696c 716171716271 7461696c NULL
+2c0000 65656162632c7461696c 6565616263 7461696c NULL
+2c0000 716c656674 716c656674 NULL NULL
+2c0000 7269676874717c7461696c 7269676874717c7461696c NULL NULL
+2c0000 616263716c6566747c7269676874717c7461696c
616263716c6566747c7269676874717c7461696c NULL NULL
+2c0000 6100622c7461696c 6162 7461696c NULL
+2c0000 EMPTY NULL NULL NULL
+2c0000 2c EMPTY EMPTY NULL
+2c0000 7c7c 7c7c NULL NULL
+2c0000 6c6566747c71756e636c6f736564 6c6566747c71756e636c6f736564 NULL
NULL
+2c0000 716c656674717c7461696c 716c656674717c7461696c NULL NULL
+2c0000 716c65667471717269676874717c7461696c
716c65667471717269676874717c7461696c NULL NULL
+2c0000 787c092071617c62717c63 787c092071617c62717c63 NULL NULL
+2c0000 787ce28083e2808371617c62717c63 787ce28083e2808371617c62717c63 NULL
NULL
+2c0000 787cc2a071617c62717c63 787cc2a071617c62717c63 NULL NULL
+2c0000 c3a9716c6566747c7269676874717c7461696c
c3a9716c6566747c7269676874717c7461696c NULL NULL
+2c0000 f09f9880716c6566747c7269676874717c7461696c
f09f9880716c6566747c7269676874717c7461696c NULL NULL
+2c0000 616263716c656674c3a9726967687471c3a97461696c
616263716c656674c3a9726967687471c3a97461696c NULL NULL
+2c0000 71c3a971c3a97461696c 71c3a971c3a97461696c NULL NULL
+2c0000 2222227822222c7461696c 222222782222 7461696c NULL
+2c0000 6520007c205c2000616120202c5c2c 65207c205c2061612020 5c EMPTY
+2c0000 20 20 NULL NULL
+2c0000 EMPTY NULL NULL NULL
+2c0000 222061205c7c20717c65005c205c71715c00617c71
222061205c7c20717c655c205c71715c617c71 NULL NULL
+2c0000 007161202c7c7161612071652c2071 716120 7c716161207165 2071
+2c0000 71007c0065657c00616100207c7c 717c65657c6161207c7c NULL NULL
+2c0000 22710022617161612020 227122617161612020 NULL NULL
+2c0000 5c7c6161617c20716520205c2c 5c7c6161617c20716520205c EMPTY
NULL
+2c0000 65 65 NULL NULL
+2c0000 00652c227c6561222200222c2222207165617c2265005c 65 227c6561222222
2222207165617c22655c
+2c0000 5c7165225c71226161 5c7165225c71226161 NULL NULL
+2c0000 7c005c2c007c22612c2c65 7c5c 7c2261 EMPTY
+2c0000 612c5c20205c7c5c7c00226520717c2c655c655c2c6122 61
5c20205c7c5c7c226520717c 655c655c
+2c0000 007c6161200022615c202061227122 7c61612022615c202061227122 NULL
NULL
+2c0000 2c5c20002c5c5c2c225c5c71 EMPTY 5c20 5c5c
+2c0000 2265652022712c712c00657120202261207c5c 226565202271 71
657120202261207c5c
+2c0000 657c5c225c0065225c65656565 657c5c225c65225c65656565 NULL
NULL
+2c0000 2c5c715c5c20225c71227c5c00656120717c7c5c61 EMPTY
5c715c5c20225c71227c5c656120717c7c5c61 NULL
+2c0000 206561612c20000065652265205c00 20656161 200065652265205c
NULL
+2c0000 00002c2c5c20615c5c00 EMPTY EMPTY 5c20615c5c
+2c0000 717c6161222c 717c616122 EMPTY NULL
+2c0000 0020226171655c5c 20226171655c5c NULL NULL
+2c0000 5c007c22 5c7c22 NULL NULL
+2c0000 0022202261 22202261 NULL NULL
+2c0000 6500617c220020 65617c2220 NULL NULL
+2c0000 20655c7c5c 20655c7c5c NULL NULL
+2c0000 20612c2c71006171 2061 EMPTY 716171
+2c0000 202222717c7c5c7120202222226565617c205c
202222717c7c5c7120202222226565617c205c NULL NULL
+2c0000 205c2c007120207c5c005c20002c22007c5c5c5c22 205c 7120207c5c5c20
227c5c5c5c22
+2c0000 7c20712200652c7c2c2c 7c20712265 7c EMPTY
+2c0000 65202c005c2061007161227c5c7c2c717c65657c61 6520
5c20617161227c5c7c 717c65657c61
+2c0000 00227c7c7c715c652c0020712c71615c 227c7c7c715c65 2071 71615c
+2c0000 007c 7c NULL NULL
+2c0000 2c0020206120617171716561612c5c20 EMPTY 2020612061717171656161
5c20
+2c0000 657161205c22 657161205c22 NULL NULL
+2c0000 655c202c65207c7165612271225c657c 655c20
65207c7165612271225c657c NULL
+2c0000 6520612022612c2261 652061202261 2261 NULL
+2c0000 2c2c71007c71615c7c22715c2271 EMPTY EMPTY 717c71615c7c22715c2271
+2c0000 EMPTY NULL NULL NULL
+2c0000 6561225c 6561225c NULL NULL
+2c0000 7c617c225c22206520 7c617c225c22206520 NULL NULL
+2c0000 EMPTY NULL NULL NULL
+2c0000 61 61 NULL NULL
+2c0000 227c00 227c NULL NULL
+2c0000 61612c00615c2c5c5c226520202c002061007c5c71 6161 615c
5c5c22652020
+2c0000 2000 20 NULL NULL
+2c0000 22005c7c5c6171657c71656120006161 225c7c5c6171657c716561206161
NULL NULL
+2c0000 617c2c 617c EMPTY NULL
+2c0000 715c612c2c61 715c61 EMPTY 61
+2c0000 2c207c2c225c2c7161 EMPTY 207c 225c
+2c0000 657c652c2c6500002200656565207c6520 657c65 EMPTY
650022656565207c6520
+2c0000 7122005c22207171612020712c2c61615c206122712c65 71225c2220717161202071
EMPTY 61615c20612271
+2c0000 2c715c71657c61716122657c EMPTY 715c71657c61716122657c NULL
+2c0000 00 EMPTY NULL NULL
+2c0000 2061 2061 NULL NULL
+2c0000 205c2000 205c20 NULL NULL
+2c0000 71655c2c207c20715c0000006500227c6161 71655c 207c20715c0065227c6161
NULL
+2c0000 5c6571 5c6571 NULL NULL
+2c0000 007c2c652c7171 7c 65 7171
+2c0000 00615c 615c NULL NULL
+2c0000 71656522006565225c00 716565226565225c NULL NULL
+2c0000 002c7165205c65227c6165716565207c716171612c2c EMPTY
7165205c65227c6165716565207c71617161 EMPTY
+2c0000 222c71206120005c612c2c 22 712061205c61 EMPTY
+2c0000 7c227c617c7c22206500712022002c7c00652c5c
7c227c617c7c222065712022 7c65 5c
+2c0000 61612c225c5c20652222205c2071222261 6161
225c5c20652222205c2071222261 NULL
+2c0000 61 61 NULL NULL
+2c0000 205c2c5c2c 205c 5c EMPTY
+2c0000 2c6500007c22 EMPTY 65007c22 NULL
+2c0000 5c7c2c65202c5c7c610022227c225c 5c7c 6520 5c7c6122227c225c
+2c0000 655c2c002c7165716561205c7100207c2c7171225c2c20 655c EMPTY
7165716561205c71207c
+2c0000 2c22 EMPTY 22 NULL
+2c0000 6571 6571 NULL NULL
+2c0000 5c205c 5c205c NULL NULL
+2c0000 65715c0071615c7c2271207c5c7161207c00207c65
65715c71615c7c2271207c5c7161207c207c65 NULL NULL
+2c0000 205c20615c220000222c615c5c2c00225c71 205c20615c220022 615c5c
225c71
+2c0000 2c7c5c20657c5c7c202c EMPTY 7c5c20657c5c7c20 EMPTY
+2c0000 5c65002c005c5c2c2220 5c65 5c5c 2220
+2c0000 71 71 NULL NULL
+2c0000 71617c7c65615c61715c2c652200 71617c7c65615c61715c 6522 NULL
+2c0000 7c7165715c5c5c612065655c 7c7165715c5c5c612065655c NULL
NULL
+2c225c 787c202071617c62717c63 787c202071617c62717c63 NULL NULL
+2c225c 7161717162712c7461696c 716171716271 7461696c NULL
+2c225c 65656162632c7461696c 6565616263 7461696c NULL
+2c225c 716c656674 716c656674 NULL NULL
+2c225c 7269676874717c7461696c 7269676874717c7461696c NULL NULL
+2c225c 616263716c6566747c7269676874717c7461696c
616263716c6566747c7269676874717c7461696c NULL NULL
+2c225c 6100622c7461696c 610062 7461696c NULL
+2c225c EMPTY NULL NULL NULL
+2c225c 2c EMPTY EMPTY NULL
+2c225c 7c7c 7c7c NULL NULL
+2c225c 6c6566747c71756e636c6f736564 6c6566747c71756e636c6f736564 NULL
NULL
+2c225c 716c656674717c7461696c 716c656674717c7461696c NULL NULL
+2c225c 716c65667471717269676874717c7461696c
716c65667471717269676874717c7461696c NULL NULL
+2c225c 787c092071617c62717c63 787c092071617c62717c63 NULL NULL
+2c225c 787ce28083e2808371617c62717c63 787ce28083e2808371617c62717c63 NULL
NULL
+2c225c 787cc2a071617c62717c63 787cc2a071617c62717c63 NULL NULL
+2c225c c3a9716c6566747c7269676874717c7461696c
c3a9716c6566747c7269676874717c7461696c NULL NULL
+2c225c f09f9880716c6566747c7269676874717c7461696c
f09f9880716c6566747c7269676874717c7461696c NULL NULL
+2c225c 616263716c656674c3a9726967687471c3a97461696c
616263716c656674c3a9726967687471c3a97461696c NULL NULL
+2c225c 71c3a971c3a97461696c 71c3a971c3a97461696c NULL NULL
+2c225c 2222227822222c7461696c NULL NULL NULL
+2c225c 6520007c205c2000616120202c5c2c 6520007c20200061612020 EMPTY EMPTY
+2c225c 20 20 NULL NULL
+2c225c EMPTY NULL NULL NULL
+2c225c 222061205c7c20717c65005c205c71715c00617c71 NULL NULL NULL
+2c225c 007161202c7c7161612071652c2071 00716120 7c716161207165 2071
+2c225c 71007c0065657c00616100207c7c 71007c0065657c00616100207c7c NULL
NULL
+2c225c 22710022617161612020 710022617161612020 NULL NULL
+2c225c 5c7c6161617c20716520205c2c 7c6161617c2071652020 EMPTY NULL
+2c225c 65 65 NULL NULL
+2c225c 00652c227c6561222200222c2222207165617c2265005c 0065 7c65612200
NULL
+2c225c 5c7165225c71226161 71652271226161 NULL NULL
+2c225c 7c005c2c007c22612c2c65 7c00 NULL NULL
+2c225c 612c5c20205c7c5c7c00226520717c2c655c655c2c6122 61
20207c7c00226520717c2c65652c61 NULL
+2c225c 007c6161200022615c202061227122 NULL NULL NULL
+2c225c 2c5c20002c5c5c2c225c5c71 EMPTY 2000 EMPTY
+2c225c 2265652022712c712c00657120202261207c5c 6565202271 71 NULL
+2c225c 657c5c225c0065225c65656565 NULL NULL NULL
+2c225c 2c5c715c5c20225c71227c5c00656120717c7c5c61 EMPTY
715c202271227c00656120717c7c61 NULL
+2c225c 206561612c20000065652265205c00 20656161 NULL NULL
+2c225c 00002c2c5c20615c5c00 0000 EMPTY 20615c00
+2c225c 717c6161222c NULL NULL NULL
+2c225c 0020226171655c5c NULL NULL NULL
+2c225c 5c007c22 NULL NULL NULL
+2c225c 0022202261 00202261 NULL NULL
+2c225c 6500617c220020 NULL NULL NULL
+2c225c 20655c7c5c 20657c NULL NULL
+2c225c 20612c2c71006171 2061 EMPTY 71006171
+2c225c 202222717c7c5c7120202222226565617c205c NULL NULL NULL
+2c225c 205c2c007120207c5c005c20002c22007c5c5c5c22 20
007120207c002000 NULL
+2c225c 7c20712200652c7c2c2c NULL NULL NULL
+2c225c 65202c005c2061007161227c5c7c2c717c65657c61 6520 NULL NULL
+2c225c 00227c7c7c715c652c0020712c71615c NULL NULL NULL
+2c225c 007c 007c NULL NULL
+2c225c 2c0020206120617171716561612c5c20 EMPTY
002020612061717171656161 20
+2c225c 657161205c22 6571612022 NULL NULL
+2c225c 655c202c65207c7165612271225c657c 6520 65207c716561227122657c
NULL
+2c225c 6520612022612c2261 6520612022612c61 NULL NULL
+2c225c 2c2c71007c71615c7c22715c2271 EMPTY EMPTY NULL
+2c225c EMPTY NULL NULL NULL
+2c225c 6561225c NULL NULL NULL
+2c225c 7c617c225c22206520 NULL NULL NULL
+2c225c EMPTY NULL NULL NULL
+2c225c 61 61 NULL NULL
+2c225c 227c00 NULL NULL NULL
+2c225c 61612c00615c2c5c5c226520202c002061007c5c71 6161 0061 NULL
+2c225c 2000 2000 NULL NULL
+2c225c 22005c7c5c6171657c71656120006161 NULL NULL NULL
+2c225c 617c2c 617c EMPTY NULL
+2c225c 715c612c2c61 7161 EMPTY 61
+2c225c 2c207c2c225c2c7161 EMPTY 207c NULL
+2c225c 657c652c2c6500002200656565207c6520 657c65 EMPTY NULL
+2c225c 7122005c22207171612020712c2c61615c206122712c65
710022207171612020712c2c616120612271 65 NULL
+2c225c 2c715c71657c61716122657c EMPTY NULL NULL
+2c225c 00 00 NULL NULL
+2c225c 2061 2061 NULL NULL
+2c225c 205c2000 202000 NULL NULL
+2c225c 71655c2c207c20715c0000006500227c6161 7165 NULL NULL
+2c225c 5c6571 6571 NULL NULL
+2c225c 007c2c652c7171 007c 65 7171
+2c225c 00615c 0061 NULL NULL
+2c225c 71656522006565225c00 716565220065652200 NULL NULL
+2c225c 002c7165205c65227c6165716565207c716171612c2c 00 NULL NULL
+2c225c 222c71206120005c612c2c NULL NULL NULL
+2c225c 7c227c617c7c22206500712022002c7c00652c5c NULL NULL NULL
+2c225c 61612c225c5c20652222205c2071222261 6161 NULL NULL
+2c225c 61 61 NULL NULL
+2c225c 205c2c5c2c 20 EMPTY EMPTY
+2c225c 2c6500007c22 EMPTY NULL NULL
+2c225c 5c7c2c65202c5c7c610022227c225c 7c 6520 NULL
+2c225c 655c2c002c7165716561205c7100207c2c7171225c2c20 65 00
7165716561207100207c
+2c225c 2c22 EMPTY NULL NULL
+2c225c 6571 6571 NULL NULL
+2c225c 5c205c 20 NULL NULL
+2c225c 65715c0071615c7c2271207c5c7161207c00207c65 NULL NULL NULL
+2c225c 205c20615c220000222c615c5c2c00225c71 2020612200002c615c2c002271
NULL NULL
+2c225c 2c7c5c20657c5c7c202c EMPTY 7c20657c7c20 EMPTY
+2c225c 5c65002c005c5c2c2220 6500 005c NULL
+2c225c 71 71 NULL NULL
+2c225c 71617c7c65615c61715c2c652200 71617c7c65616171 NULL NULL
+2c225c 7c7165715c5c5c612065655c 7c7165715c61206565 NULL NULL
+c3a97165 787c202071617c62717c63 787c202071617c62717c63 NULL NULL
+c3a97165 7161717162712c7461696c 617162712c7461696c NULL NULL
+c3a97165 65656162632c7461696c 6162632c7461696c NULL NULL
+c3a97165 716c656674 NULL NULL NULL
+c3a97165 7269676874717c7461696c NULL NULL NULL
+c3a97165 616263716c6566747c7269676874717c7461696c
616263716c66747c7269676874717c7461696c NULL NULL
+c3a97165 6100622c7461696c 6100622c7461696c NULL NULL
+c3a97165 EMPTY NULL NULL NULL
+c3a97165 2c 2c NULL NULL
+c3a97165 7c7c 7c7c NULL NULL
+c3a97165 6c6566747c71756e636c6f736564 NULL NULL NULL
+c3a97165 716c656674717c7461696c 6c6674717c7461696c NULL NULL
+c3a97165 716c65667471717269676874717c7461696c
6c6674717269676874717c7461696c NULL NULL
+c3a97165 787c092071617c62717c63 787c092071617c62717c63 NULL NULL
+c3a97165 787ce28083e2808371617c62717c63 787ce28083e2808371617c62717c63
NULL NULL
+c3a97165 787cc2a071617c62717c63 787cc2a071617c62717c63 NULL NULL
+c3a97165 c3a9716c6566747c7269676874717c7461696c EMPTY
6c66747c7269676874717c7461696c NULL
+c3a97165 f09f9880716c6566747c7269676874717c7461696c
f09f98806c66747c7269676874717c7461696c NULL NULL
+c3a97165 616263716c656674c3a9726967687471c3a97461696c
616263716c6674c3a97269676874 7461696c NULL
+c3a97165 71c3a971c3a97461696c c3a9 7461696c NULL
+c3a97165 2222227822222c7461696c 2222227822222c7461696c NULL NULL
+c3a97165 6520007c205c2000616120202c5c2c 20007c205c2000616120202c5c2c
NULL NULL
+c3a97165 20 20 NULL NULL
+c3a97165 EMPTY NULL NULL NULL
+c3a97165 222061205c7c20717c65005c205c71715c00617c71
222061205c7c20717c005c205c715c00617c NULL NULL
+c3a97165 007161202c7c7161612071652c2071 0061202c7c71616120712c20
NULL NULL
+c3a97165 71007c0065657c00616100207c7c NULL NULL NULL
+c3a97165 22710022617161612020 220022617161612020 NULL NULL
+c3a97165 5c7c6161617c20716520205c2c NULL NULL NULL
+c3a97165 65 EMPTY NULL NULL
+c3a97165 00652c227c6561222200222c2222207165617c2265005c NULL NULL
NULL
+c3a97165 5c7165225c71226161 5c225c71226161 NULL NULL
+c3a97165 7c005c2c007c22612c2c65 7c005c2c007c22612c2c NULL NULL
+c3a97165 612c5c20205c7c5c7c00226520717c2c655c655c2c6122 NULL NULL
NULL
+c3a97165 007c6161200022615c202061227122 NULL NULL NULL
+c3a97165 2c5c20002c5c5c2c225c5c71 NULL NULL NULL
+c3a97165 2265652022712c712c00657120202261207c5c
22652022712c712c007120202261207c5c NULL NULL
+c3a97165 657c5c225c0065225c65656565 7c5c225c00225c6565 NULL
NULL
+c3a97165 2c5c715c5c20225c71227c5c00656120717c7c5c61 NULL NULL
NULL
+c3a97165 206561612c20000065652265205c00 2061612c2000006522205c00
NULL NULL
+c3a97165 00002c2c5c20615c5c00 00002c2c5c20615c5c00 NULL NULL
+c3a97165 717c6161222c NULL NULL NULL
+c3a97165 0020226171655c5c NULL NULL NULL
+c3a97165 5c007c22 5c007c22 NULL NULL
+c3a97165 0022202261 0022202261 NULL NULL
+c3a97165 6500617c220020 00617c220020 NULL NULL
+c3a97165 20655c7c5c 205c7c5c NULL NULL
+c3a97165 20612c2c71006171 20612c2c710061 NULL NULL
+c3a97165 202222717c7c5c7120202222226565617c205c
202222717c7c5c71202022222265617c205c NULL NULL
+c3a97165 205c2c007120207c5c005c20002c22007c5c5c5c22 NULL NULL
NULL
+c3a97165 7c20712200652c7c2c2c NULL NULL NULL
+c3a97165 65202c005c2061007161227c5c7c2c717c65657c61
202c005c2061007161227c5c7c2c717c657c61 NULL NULL
+c3a97165 00227c7c7c715c652c0020712c71615c NULL NULL NULL
+c3a97165 007c 007c NULL NULL
+c3a97165 2c0020206120617171716561612c5c20 NULL NULL NULL
+c3a97165 657161205c22 NULL NULL NULL
+c3a97165 655c202c65207c7165612271225c657c
5c202c207c71612271225c7c NULL NULL
+c3a97165 6520612022612c2261 20612022612c2261 NULL NULL
+c3a97165 2c2c71007c71615c7c22715c2271 2c2c007c71615c7c22715c22
NULL NULL
+c3a97165 EMPTY NULL NULL NULL
+c3a97165 6561225c 61225c NULL NULL
+c3a97165 7c617c225c22206520 7c617c225c222020 NULL NULL
+c3a97165 EMPTY NULL NULL NULL
+c3a97165 61 61 NULL NULL
+c3a97165 227c00 227c00 NULL NULL
+c3a97165 61612c00615c2c5c5c226520202c002061007c5c71 NULL NULL
NULL
+c3a97165 2000 2000 NULL NULL
+c3a97165 22005c7c5c6171657c71656120006161
22005c7c5c61717c716120006161 NULL NULL
+c3a97165 617c2c 617c2c NULL NULL
+c3a97165 715c612c2c61 NULL NULL NULL
+c3a97165 2c207c2c225c2c7161 NULL NULL NULL
+c3a97165 657c652c2c6500002200656565207c6520 7c2c2c0000220065207c20
NULL NULL
+c3a97165 7122005c22207171612020712c2c61615c206122712c65 NULL NULL
NULL
+c3a97165 2c715c71657c61716122657c NULL NULL NULL
+c3a97165 00 00 NULL NULL
+c3a97165 2061 2061 NULL NULL
+c3a97165 205c2000 205c2000 NULL NULL
+c3a97165 71655c2c207c20715c0000006500227c6161
5c2c207c20715c00000000227c6161 NULL NULL
+c3a97165 5c6571 5c71 NULL NULL
+c3a97165 007c2c652c7171 007c2c2c71 NULL NULL
+c3a97165 00615c 00615c NULL NULL
+c3a97165 71656522006565225c00 NULL NULL NULL
+c3a97165 002c7165205c65227c6165716565207c716171612c2c NULL NULL
NULL
+c3a97165 222c71206120005c612c2c NULL NULL NULL
+c3a97165 7c227c617c7c22206500712022002c7c00652c5c NULL NULL
NULL
+c3a97165 61612c225c5c20652222205c2071222261 NULL NULL NULL
+c3a97165 61 61 NULL NULL
+c3a97165 205c2c5c2c 205c2c5c2c NULL NULL
+c3a97165 2c6500007c22 2c00007c22 NULL NULL
+c3a97165 5c7c2c65202c5c7c610022227c225c 5c7c2c202c5c7c610022227c225c
NULL NULL
+c3a97165 655c2c002c7165716561205c7100207c2c7171225c2c20
5c2c002c717161205c7100207c2c71225c2c20 NULL NULL
+c3a97165 2c22 2c22 NULL NULL
+c3a97165 6571 NULL NULL NULL
+c3a97165 5c205c 5c205c NULL NULL
+c3a97165 65715c0071615c7c2271207c5c7161207c00207c65
5c0071615c7c2271207c5c7161207c00207c NULL NULL
+c3a97165 205c20615c220000222c615c5c2c00225c71 NULL NULL NULL
+c3a97165 2c7c5c20657c5c7c202c 2c7c5c207c5c7c202c NULL NULL
+c3a97165 5c65002c005c5c2c2220 5c002c005c5c2c2220 NULL NULL
+c3a97165 71 NULL NULL NULL
+c3a97165 71617c7c65615c61715c2c652200 617c7c615c61715c2c2200 NULL
NULL
+c3a97165 7c7165715c5c5c612065655c NULL NULL NULL
diff --git
a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ScanNodePropertyKeys.java
b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ScanNodePropertyKeys.java
index fff00e0cbec..ab27eea6012 100644
---
a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ScanNodePropertyKeys.java
+++
b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/scan/ScanNodePropertyKeys.java
@@ -132,6 +132,9 @@ public final class ScanNodePropertyKeys {
/** Quote character enclosing a field; a single character. */
public static final String TEXT_ENCLOSE = TEXT_PROPERTY_PREFIX + "enclose";
+ /** {@code "true"} for Hive OpenCSV field states and physical record
boundaries. */
+ public static final String TEXT_HIVE_OPEN_CSV = TEXT_PROPERTY_PREFIX +
"hive_open_csv";
+
/** {@code "true"} to strip the enclosing quotes from field values. */
public static final String TEXT_TRIM_DOUBLE_QUOTES = TEXT_PROPERTY_PREFIX
+ "trim_double_quotes";
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
index 22ba216e0a2..9def858c616 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
+++
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.connector.spi;
import org.apache.doris.connector.spi.handle.ConnectorColumnHandle;
import org.apache.doris.connector.spi.handle.ConnectorWriteHandle;
import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider;
+import org.apache.doris.connector.spi.scan.ScanNodePropertyKeys;
import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider;
import org.junit.jupiter.api.Assertions;
@@ -29,7 +30,9 @@ import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
+import java.lang.reflect.Field;
import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
@@ -45,8 +48,9 @@ import java.util.TreeSet;
* on a plugin author and nothing fails when either surface quietly changes.
The plugin API version in
* {@code <connector.plugin.api.version>} is the contract that says which FE a
given plugin may load into,
* and the rule attached to it is blunt: <em>any</em> change to the surface
below — adding a type, method, or
- * enum constant just as much as removing or re-signing one — is a MAJOR
change. No unit test can prove somebody
- * bumped the property (a test sees only the current state, never the delta),
so this is a speed bump, not a
+ * enum constant or engine-read property key just as much as removing or
re-signing one — is a MAJOR change.
+ * No unit test can prove somebody bumped the property (a test sees only the
current state, never the delta),
+ * so this is a speed bump, not a
* gate: it makes the change visible in review, in the same commit, with the
reason spelled out in the
* failure message.
*
@@ -81,8 +85,8 @@ public class ConnectorPluginSurfaceTest {
Assertions.assertNotNull(in, "missing connector plugin API version
resource");
version.load(in);
}
- // Latest-schema publication is explicit in major 8; older engines
cannot honor the opt-in contract.
- Assertions.assertEquals("8.0", version.getProperty("api.version"));
+ // OpenCSV scan properties require major 9: inlined keys cannot fail
JVM linkage on an older FE.
+ Assertions.assertEquals("9.0", version.getProperty("api.version"));
}
/** Root entry points plus provider/handle types returned to connector
plugins. */
@@ -106,7 +110,7 @@ public class ConnectorPluginSurfaceTest {
Arrays.asList(ConnectorCapability.class);
@Test
- public void pluginApiSurfaceMatchesRecordedBaseline() throws IOException {
+ public void pluginApiSurfaceMatchesRecordedBaseline() throws IOException,
IllegalAccessException {
TreeSet<String> actual = renderSurface();
TreeSet<String> expected = readBaseline();
@@ -130,8 +134,15 @@ public class ConnectorPluginSurfaceTest {
* happens to declare it: what matters is what a plugin can call on the
type it was handed, so moving a
* default method up or down a super-interface chain is not by itself a
surface change.
*/
- private static TreeSet<String> renderSurface() {
+ private static TreeSet<String> renderSurface() throws
IllegalAccessException {
TreeSet<String> rendered = new TreeSet<>();
+ // String constants are inlined into plugins, so both their names and
wire values are API surface.
+ for (Field field : ScanNodePropertyKeys.class.getFields()) {
+ Assertions.assertEquals(String.class, field.getType());
+ Assertions.assertTrue(Modifier.isStatic(field.getModifiers()) &&
Modifier.isFinal(field.getModifiers()));
+ rendered.add(ScanNodePropertyKeys.class.getName() + "#field:" +
field.getName()
+ + ":" + field.getType().getTypeName() + "=" +
field.get(null));
+ }
for (Class<? extends Enum<?>> frozen : FROZEN_ENUM_TYPES) {
for (Enum<?> constant : frozen.getEnumConstants()) {
rendered.add(frozen.getName() + "#enum:" + constant.name());
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
index 663b6a8b7fd..f3d8d8cb237 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
+++
b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
@@ -121,6 +121,30 @@
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTabl
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsSystemTableTimeTravel():boolean
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#supportsTableSample():boolean
org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider#usesHiveParquetInt96TimeZone():boolean
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:FILE_FORMAT_TYPE:java.lang.String=file_format_type
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:LOCATION_PREFIX:java.lang.String=location.
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:PATH_PARTITION_KEYS:java.lang.String=path_partition_keys
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:REMOTE_QUERY:java.lang.String=query
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:REQUIRED_CURRENT_BACKEND_SEMANTICS:java.lang.String=required_current_backend_semantics
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:SYNTHETIC_ALL_CONJUNCTS_PUSHED:java.lang.String=__all_conjuncts_pushed
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:SYNTHETIC_EXPLAIN_VERBOSE:java.lang.String=__explain_verbose
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:SYNTHETIC_NATIVE_READ_SPLITS:java.lang.String=__native_read_splits
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:SYNTHETIC_PUSHDOWN_LIMIT:java.lang.String=__pushdown_limit
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:SYNTHETIC_TOTAL_READ_SPLITS:java.lang.String=__total_read_splits
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_COLLECTION_DELIMITER:java.lang.String=hive.text.collection_delimiter
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_COLUMN_SEPARATOR:java.lang.String=hive.text.column_separator
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_ENCLOSE:java.lang.String=hive.text.enclose
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_ESCAPE:java.lang.String=hive.text.escape
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_HIVE_OPEN_CSV:java.lang.String=hive.text.hive_open_csv
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_IS_JSON:java.lang.String=hive.text.is_json
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_LINE_DELIMITER:java.lang.String=hive.text.line_delimiter
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_MAPKV_DELIMITER:java.lang.String=hive.text.mapkv_delimiter
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_NULL_FORMAT:java.lang.String=hive.text.null_format
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_OPENX_IGNORE_MALFORMED:java.lang.String=hive.text.openx_ignore_malformed
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_PROPERTY_PREFIX:java.lang.String=hive.text.
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_SERDE_LIB:java.lang.String=hive.text.serde_lib
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_SKIP_LINES:java.lang.String=hive.text.skip_lines
+org.apache.doris.connector.spi.scan.ScanNodePropertyKeys#field:TEXT_TRIM_DOUBLE_QUOTES:java.lang.String=hive.text.trim_double_quotes
org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#appendExplainInfo(java.lang.StringBuilder,java.lang.String,org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorWriteHandle):void
org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getSyntheticWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle):java.util.List
org.apache.doris.connector.spi.write.ConnectorWritePlanProvider#getWriteColumns(org.apache.doris.connector.spi.ConnectorSession,org.apache.doris.connector.spi.handle.ConnectorTableHandle,java.util.Optional):java.util.Optional
diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml
index f7449826034..c00ef144e59 100644
--- a/fe/fe-connector/pom.xml
+++ b/fe/fe-connector/pom.xml
@@ -55,7 +55,7 @@ under the License.
of the latter two means bumping this property as well (and
fe-extension-spi means bumping
all five families).
-->
- <connector.plugin.api.version>8.0</connector.plugin.api.version>
+ <connector.plugin.api.version>9.0</connector.plugin.api.version>
</properties>
<modules>
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java
index f3c24c0bb7f..dca931a3255 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java
@@ -1102,6 +1102,14 @@ public class PluginDrivenScanNode extends
FileQueryScanNode {
attrs.setTrimDoubleQuotes(true);
}
+ if ("true".equals(props.get(ScanNodePropertyKeys.TEXT_HIVE_OPEN_CSV)))
{
+ // The optional wire field alone cannot fence old readers during a
rolling upgrade.
+ if (Config.be_exec_version <
Config.HIVE_OPEN_CSV_MIN_BE_EXEC_VERSION) {
+ throw new UserException("Hive OpenCSVSerde requires backend
execution version "
+ + Config.HIVE_OPEN_CSV_MIN_BE_EXEC_VERSION + " or
newer during rolling upgrade");
+ }
+ attrs.setHiveOpenCsv(true);
+ }
attrs.setTextParams(textParams);
attrs.setHeaderType("");
attrs.setEnableTextValidateUtf8(sessionVariable.enableTextValidateUtf8);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCsvPropertiesTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCsvPropertiesTest.java
new file mode 100644
index 00000000000..503cb68df41
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCsvPropertiesTest.java
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.scan;
+
+import org.apache.doris.common.Config;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.connector.hive.HiveTextProperties;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.connector.spi.scan.ScanNodePropertyKeys;
+import org.apache.doris.qe.SessionVariable;
+import org.apache.doris.system.Backend;
+import org.apache.doris.thrift.TFileAttributes;
+import org.apache.doris.thrift.TFileTextScanRangeParams;
+
+import org.apache.thrift.TDeserializer;
+import org.apache.thrift.TSerializer;
+import org.apache.thrift.protocol.TCompactProtocol;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+class PluginDrivenScanNodeCsvPropertiesTest {
+ private static final String CSV_SERDE =
"org.apache.hadoop.hive.serde2.OpenCSVSerde";
+
+ @Test
+ void testNormalizedCharactersSurviveScanNodeAndThrift() throws Exception {
+ Map<String, String> properties = Map.of("separatorChar", "ss",
"quoteChar", "qq", "escapeChar", "ee",
+ "line.delim", "|");
+ assertWireProperties(toWire(Map.of(), properties), "s", 'q', 'e',
false);
+ assertWireProperties(toWire(properties, Map.of("line.delim", "|")),
"s", 'q', 'e', false);
+ }
+
+ @Test
+ void testEffectiveTablePropertiesOverrideInvalidSerdeProperties() throws
Exception {
+ assertWireProperties(toWire(Map.of("separatorChar", "", "quoteChar",
"é", "escapeChar", ""),
+ Map.of("separatorChar", "ss", "quoteChar", "qq", "escapeChar",
"ee")),
+ "s", 'q', 'e', false);
+ }
+
+ @Test
+ void testDefaultsAndNormalizedDoubleQuote() throws Exception {
+ assertWireProperties(toWire(Map.of(), Map.of()), ",", '"', '\\', true);
+ assertWireProperties(toWire(Map.of(),
+ Map.of("separatorChar", "99", "quoteChar", "\"suffix",
"escapeChar", "eé")),
+ "9", '"', 'e', true);
+ }
+
+ @Test
+ void testUtf8SeparatorRemainsACompleteCharacter() throws Exception {
+ assertWireProperties(toWire(Map.of(), Map.of("separatorChar",
"ésuffix")), "é", '"', '\\', true);
+ }
+
+ @Test
+ void testInvalidCharactersFailBeforeThriftConstruction() {
+ for (String key : new String[] {"separatorChar", "quoteChar",
"escapeChar"}) {
+ Assertions.assertThrows(DorisConnectorException.class, () ->
toWire(Map.of(), Map.of(key, "")));
+ }
+ for (String key : new String[] {"quoteChar", "escapeChar"}) {
+ Assertions.assertThrows(DorisConnectorException.class, () ->
toWire(Map.of(), Map.of(key, "é")));
+ }
+ Assertions.assertThrows(DorisConnectorException.class,
+ () -> toWire(Map.of(), Map.of("separatorChar", "😀")));
+ }
+
+ @Test
+ void testDefaultEscapeSentinelSurvivesThriftAsBackslash() throws Exception
{
+ for (String quote : new String[] {"\"", "q"}) {
+ Map<String, String> properties = Map.of("quoteChar", quote,
"escapeChar", "\"suffix");
+ assertWireProperties(toWire(Map.of(), properties), ",",
quote.charAt(0), '\\', "\"".equals(quote));
+ assertWireProperties(toWire(properties, Map.of()), ",",
quote.charAt(0), '\\', "\"".equals(quote));
+ }
+ }
+
+ @Test
+ void testCharacterConflictsFailBeforeThriftConstruction() {
+ for (Map<String, String> properties : List.of(
+ Map.of("separatorChar", "|", "quoteChar", "|"),
+ Map.of("separatorChar", "|", "escapeChar", "|"),
+ Map.of("quoteChar", "q", "escapeChar", "q"),
+ Map.of("separatorChar", "\\", "escapeChar", "\""),
+ Map.of("separatorChar", "\0"))) {
+ Assertions.assertThrows(DorisConnectorException.class, () ->
toWire(Map.of(), properties));
+ Assertions.assertThrows(DorisConnectorException.class, () ->
toWire(properties, Map.of()));
+ }
+ }
+
+ @Test
+ void testDisabledQuoteAndEscapeSurviveThrift() throws Exception {
+ Map<String, String> properties = Map.of("quoteChar", "\0",
"escapeChar", "\0");
+ assertWireProperties(toWire(Map.of(), properties), ",", '\0', '\0',
false);
+ assertWireProperties(toWire(properties, Map.of()), ",", '\0', '\0',
false);
+ }
+
+ @Test
+ void testOtherHiveSerdesDoNotEnableOpenCsvParsing() throws Exception {
+ for (String serde : List.of(HiveTextProperties.HIVE_TEXT_SERDE,
HiveTextProperties.HIVE_JSON_SERDE)) {
+ TFileAttributes attributes =
propertiesToWire(HiveTextProperties.extract(serde, Map.of(), Map.of()));
+ Assertions.assertFalse(attributes.isSetHiveOpenCsv());
+ Assertions.assertFalse(attributes.isHiveOpenCsv());
+ }
+ }
+
+ @Test
+ void testOpenCsvRejectsOldExecutionVersion() {
+ int original = Config.be_exec_version;
+ try {
+ Config.be_exec_version = 14;
+ UserException error = Assertions.assertThrows(UserException.class,
() -> toWire(Map.of(), Map.of()));
+ Assertions.assertTrue(error.getMessage().contains("OpenCSV"));
+ Assertions.assertTrue(error.getMessage().contains("15"));
+ } finally {
+ Config.be_exec_version = original;
+ }
+ }
+
+ @Test
+ void testOpenCsvAcceptsSupportingExecutionVersion() throws Exception {
+ int original = Config.be_exec_version;
+ try {
+ Config.be_exec_version = 15;
+ Assertions.assertTrue(toWire(Map.of(), Map.of()).isHiveOpenCsv());
+ } finally {
+ Config.be_exec_version = original;
+ }
+ }
+
+ @Test
+ void testAbsentSemanticFlagRetainsLegacyWireContract() throws Exception {
+ int original = Config.be_exec_version;
+ try {
+ Config.be_exec_version = 14;
+ Map<String, String> legacy = new
HashMap<>(HiveTextProperties.extract(CSV_SERDE, Map.of(), Map.of()));
+ legacy.remove(ScanNodePropertyKeys.TEXT_HIVE_OPEN_CSV);
+
legacy.remove(ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS);
+ TFileAttributes attributes = propertiesToWire(legacy);
+ Assertions.assertFalse(attributes.isSetHiveOpenCsv());
+ Assertions.assertFalse(attributes.isHiveOpenCsv());
+ for (String serde : List.of(HiveTextProperties.HIVE_TEXT_SERDE,
HiveTextProperties.HIVE_JSON_SERDE)) {
+ Map<String, String> properties =
HiveTextProperties.extract(serde, Map.of(), Map.of());
+
Assertions.assertFalse(properties.containsKey(ScanNodePropertyKeys.REQUIRED_CURRENT_BACKEND_SEMANTICS));
+
Assertions.assertFalse(propertiesToWire(properties).isSetHiveOpenCsv());
+ }
+ } finally {
+ Config.be_exec_version = original;
+ }
+ }
+
+ @Test
+ void testOpenCsvRejectsSmoothUpgradeSourceBeforeScheduling() {
+ int original = Config.be_exec_version;
+ try {
+ Config.be_exec_version = 15;
+ PluginDrivenScanNode node =
Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS);
+ Deencapsulation.setField(node, "scanNodeProperties",
HiveTextProperties.extract(CSV_SERDE, Map.of(), Map.of()));
+ Backend source = new Backend(7L, "127.0.0.1", 9050);
+ source.setSmoothUpgradeSrc(true);
+ FederationBackendPolicy policy =
Mockito.mock(FederationBackendPolicy.class);
+ Mockito.when(policy.getBackends()).thenReturn(List.of(new
Backend(8L, "127.0.0.1", 9051), source));
+ Deencapsulation.setField(node, "backendPolicy", policy);
+ UserException error = Assertions.assertThrows(UserException.class,
node::createScanRangeLocations);
+ Assertions.assertTrue(error.getMessage().contains("OpenCSV"));
+ Assertions.assertTrue(error.getMessage().contains("smooth upgrade
source"));
+ Assertions.assertTrue(error.getMessage().contains("backend 7"));
+ } finally {
+ Config.be_exec_version = original;
+ }
+ }
+
+ private static TFileAttributes toWire(Map<String, String> serdeProperties,
+ Map<String, String> tableProperties) throws Exception {
+ return propertiesToWire(HiveTextProperties.extract(CSV_SERDE,
serdeProperties, tableProperties));
+ }
+
+ private static TFileAttributes propertiesToWire(Map<String, String>
properties) throws Exception {
+ PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class,
Mockito.CALLS_REAL_METHODS);
+ Deencapsulation.setField(node, "scanNodeProperties", properties);
+ Deencapsulation.setField(node, "sessionVariable", new
SessionVariable());
+ // Exercise the real scan-node consumer and wire encoding: map
assertions cannot detect byte truncation.
+ TFileAttributes attributes = node.getFileAttributes();
+ TFileAttributes decoded = new TFileAttributes();
+ new TDeserializer(new TCompactProtocol.Factory()).deserialize(decoded,
+ new TSerializer(new
TCompactProtocol.Factory()).serialize(attributes));
+ return decoded;
+ }
+
+ private static void assertWireProperties(TFileAttributes attributes,
String separator,
+ char quote, char escape, boolean trim) {
+ assertWireProperties(attributes, separator, quote, escape, trim, "\n");
+ }
+
+ private static void assertWireProperties(TFileAttributes attributes,
String separator,
+ char quote, char escape, boolean trim, String lineDelimiter) {
+ Assertions.assertTrue(attributes.isHiveOpenCsv());
+ TFileTextScanRangeParams text = attributes.getTextParams();
+ Assertions.assertEquals(separator, text.getColumnSeparator());
+ Assertions.assertEquals(lineDelimiter, text.getLineDelimiter());
+ Assertions.assertTrue(text.isSetEnclose());
+ Assertions.assertEquals((byte) quote, text.getEnclose());
+ Assertions.assertTrue(text.isSetEscape());
+ Assertions.assertEquals((byte) escape, text.getEscape());
+ Assertions.assertEquals("", text.getNullFormat());
+ Assertions.assertEquals(trim, attributes.isTrimDoubleQuotes());
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/pluginapiversion/PluginApiVersionWiringTest.java
b/fe/fe-core/src/test/java/org/apache/doris/pluginapiversion/PluginApiVersionWiringTest.java
index b8fe0b2d634..2ae56fb6736 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/pluginapiversion/PluginApiVersionWiringTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/pluginapiversion/PluginApiVersionWiringTest.java
@@ -107,6 +107,14 @@ public class PluginApiVersionWiringTest {
"a plugin built against another major must not become a
routable catalog type");
}
+ @Test
+ public void connectorPluginWithoutOpenCsvContractIsRefused() throws
IOException {
+ ConnectorPluginManager manager = new ConnectorPluginManager();
+
manager.loadPlugins(Collections.singletonList(connectorPluginRoot("8.0")));
+
Assertions.assertFalse(manager.getRegisteredTypes().contains("version_probe"),
+ "API 8 plugins omit the OpenCSV semantic flag and must not
load on an API 9 engine");
+ }
+
@Test
public void connectorPluginDeclaringNothingIsRefused() throws IOException {
// The regression this whole change exists for: before, a plugin that
said nothing about its API
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 8e443ab0f03..2bb119c149c 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -313,6 +313,10 @@ struct TFileAttributes {
// org.openx.data.jsonserde.JsonSerDe
13: optional bool openx_json_ignore_malformed = false;
+ // Hive OpenCSVSerde has different field states and physical record
boundaries from load CSV.
+ // Requires BE execution version >= 15 and excludes smooth-upgrade source
backends.
+ 14: optional bool hive_open_csv = false;
+
// for cloud copy into
1001: optional bool ignore_csv_redundant_col;
}
diff --git
a/regression-test/suites/external_table_p0/hive/test_csv_table_properties.groovy
b/regression-test/suites/external_table_p0/hive/test_csv_table_properties.groovy
new file mode 100644
index 00000000000..178f8cfad14
--- /dev/null
+++
b/regression-test/suites/external_table_p0/hive/test_csv_table_properties.groovy
@@ -0,0 +1,266 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_csv_table_properties",
"p0,external,hive,external_docker,external_docker_hive") {
+ String enabled = context.config.otherConfigs.get("enableHiveTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable Hive test.")
+ return
+ }
+
+ def checkV2 = { Closure check ->
+ def originalScannerV2 = sql("SHOW VARIABLES LIKE
'enable_file_scanner_v2'")[0][1]
+ try {
+ sql "SET enable_file_scanner_v2 = true"
+ check()
+ } finally {
+ sql "SET enable_file_scanner_v2 = ${originalScannerV2}"
+ }
+ }
+ setHivePrefix("hive3")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String hmsPort = context.config.otherConfigs.get("hive3HmsPort")
+ hive_docker "CREATE DATABASE IF NOT EXISTS csv_table_properties_db"
+ hive_docker "DROP TABLE IF EXISTS csv_table_properties_db.source_rows"
+ hive_docker """
+ CREATE TABLE csv_table_properties_db.source_rows (
+ row_id STRING, label STRING, payload STRING, group_id INT
+ ) STORED AS PARQUET
+ """
+ hive_docker """
+ INSERT INTO csv_table_properties_db.source_rows VALUES
+ ('1', 'Alpha', 'quiet sequoias, escape q and e', 1),
+ ('2', 'Omega', 'edge | requests, end', 4),
+ ('3', 'Beta', '"literal quotes", ss qq ee', 4),
+ ('4', 'Empty', '', 1)
+ """
+ hive_docker "SET hive.exec.dynamic.partition=true"
+ hive_docker "SET hive.exec.dynamic.partition.mode=nonstrict"
+
+ sql "DROP CATALOG IF EXISTS csv_table_properties_catalog"
+ sql """
+ CREATE CATALOG csv_table_properties_catalog PROPERTIES (
+ 'type' = 'hms',
+ 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hmsPort}'
+ )
+ """
+ sql "USE csv_table_properties_catalog.csv_table_properties_db"
+
+ // These are the four physical layouts used by CSV CTAS/INSERT product
tests. Partition
+ // values live outside the file, so a delimiter bug can corrupt data
columns while they stay correct.
+ def layouts = [
+ [table: "csv_three_columns", columns: "row_id STRING, label STRING,
payload STRING",
+ projection: "row_id, label, payload", aggregate: "max(label),
max(payload)", partitioned: false],
+ [table: "csv_partitioned_ids", columns: "row_id STRING, label STRING",
+ projection: "row_id, label, group_id", aggregate: "max(label),
max(group_id)", partitioned: true],
+ [table: "csv_two_columns", columns: "label STRING, payload STRING",
+ projection: "label, payload", aggregate: "max(label),
max(payload)", partitioned: false],
+ [table: "csv_partitioned_payload", columns: "label STRING, payload
STRING",
+ projection: "label, payload, group_id", aggregate: "max(label),
max(payload), max(group_id)",
+ partitioned: true]
+ ]
+ for (def layout : layouts) {
+ hive_docker "DROP TABLE IF EXISTS
csv_table_properties_db.${layout.table}"
+ // Keep character settings ONLY in TBLPROPERTIES: SERDEPROPERTIES
would hide the lookup bug.
+ hive_docker """
+ CREATE TABLE csv_table_properties_db.${layout.table}
(${layout.columns})
+ ${layout.partitioned ? 'PARTITIONED BY (group_id INT)' : ''}
+ ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
+ STORED AS TEXTFILE
+ TBLPROPERTIES ('separatorChar'='s', 'quoteChar'='q',
'escapeChar'='e')
+ """
+ hive_docker """
+ INSERT INTO csv_table_properties_db.${layout.table}
+ ${layout.partitioned ? 'PARTITION (group_id)' : ''}
+ SELECT ${layout.projection} FROM
csv_table_properties_db.source_rows
+ """
+
+ // Hive's text writer honors line.delim even though its CSV reader
ignores it. Set this only
+ // after INSERT so the fixture keeps newline records and tests a
read-side metadata override.
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.${layout.table} SET
TBLPROPERTIES ('line.delim'='|')
+ """
+ def sourceRows = hive_docker """
+ SELECT ${layout.projection} FROM
csv_table_properties_db.source_rows ORDER BY label
+ """
+ def hiveRows = hive_docker """
+ SELECT ${layout.projection} FROM
csv_table_properties_db.${layout.table} ORDER BY label
+ """
+ assertEquals(sourceRows, hiveRows, "Hive CSV fixture must preserve the
source rows: ${layout.table}")
+
+ sql "REFRESH DATABASE
csv_table_properties_catalog.csv_table_properties_db"
+
+ // Compare against the source data in Hive, not another CSV read with
the same delimiter bug.
+ def queries = [
+ "SELECT ${layout.projection} FROM %s ORDER BY label",
+ "SELECT ${layout.aggregate} FROM %s",
+ "SELECT ${layout.projection} FROM %s WHERE label = 'Omega' ORDER
BY label"
+ ]
+ // The same files must remain readable after a metadata-only change to
multi-character values:
+ // OpenCSVSerde takes the first Java character of each property.
+ for (boolean multiCharacter : [false, true]) {
+ if (multiCharacter) {
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.${layout.table} SET
TBLPROPERTIES (
+ 'separatorChar'='ss', 'quoteChar'='qq',
'escapeChar'='ee'
+ )
+ """
+ sql "REFRESH DATABASE
csv_table_properties_catalog.csv_table_properties_db"
+ }
+ for (String query : queries) {
+ def expected = hive_docker(String.format(query,
"csv_table_properties_db.source_rows"))
+ checkV2 {
+ def actual = sql(String.format(query, layout.table))
+ assertEquals(expected, actual)
+ }
+ }
+ }
+ }
+
+ // Hive accepts these characters, but Doris must reject them before
truncating their UTF-8 bytes.
+ // Empty values are covered by unit tests: Hive itself rejects them while
validating ALTER TABLE.
+ for (def invalid : [
+ [key: "quoteChar", value: "é", message: "the first character must be
ASCII"],
+ [key: "escapeChar", value: "é", message: "the first character must be
ASCII"]
+ ]) {
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.csv_two_columns
+ SET TBLPROPERTIES ('${invalid.key}'='${invalid.value}')
+ """
+ sql "REFRESH DATABASE
csv_table_properties_catalog.csv_table_properties_db"
+ test {
+ sql "SELECT label, payload FROM csv_two_columns ORDER BY label"
+ exception "OpenCSVSerde property '${invalid.key}':
${invalid.message}"
+ }
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.csv_two_columns SET
TBLPROPERTIES (
+ 'separatorChar'='s', 'quoteChar'='q', 'escapeChar'='e'
+ )
+ """
+ }
+ // Write with an explicit backslash, then change only the metadata to
Hive's double-quote sentinel.
+ // This leaves backslash-escaped embedded quotes in the file and exercises
Hive's reader constructor choice.
+ String sqlBackslash = "\\\\"
+ for (boolean tableProperties : [true, false]) {
+ String table = tableProperties ? "csv_default_escape_table" :
"csv_default_escape_serde"
+ hive_docker "DROP TABLE IF EXISTS csv_table_properties_db.${table}"
+ hive_docker """
+ CREATE TABLE csv_table_properties_db.${table} (label STRING,
payload STRING)
+ ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
+ ${tableProperties ? '' : "WITH SERDEPROPERTIES
('escapeChar'='${sqlBackslash}')"}
+ STORED AS TEXTFILE
+ ${tableProperties ? "TBLPROPERTIES
('escapeChar'='${sqlBackslash}')" : ''}
+ """
+ hive_docker """
+ INSERT INTO csv_table_properties_db.${table}
+ SELECT label, payload FROM csv_table_properties_db.source_rows
+ """
+ for (String sentinel : ['"', '"suffix']) {
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.${table}
+ SET ${tableProperties ? 'TBLPROPERTIES' : 'SERDEPROPERTIES'}
('escapeChar'='${sentinel}')
+ """
+ sql "REFRESH DATABASE
csv_table_properties_catalog.csv_table_properties_db"
+ def expected = hive_docker "SELECT label, payload FROM
csv_table_properties_db.source_rows ORDER BY label"
+ def hiveRows = hive_docker "SELECT label, payload FROM
csv_table_properties_db.${table} ORDER BY label"
+ assertEquals(expected, hiveRows)
+ checkV2 {
+ assertEquals(hiveRows, sql("SELECT label, payload FROM
${table} ORDER BY label"))
+ }
+ }
+ }
+
+ // Write raw single-column TEXTFILE rows before switching SerDe. The CSV
writer would normalize
+ // embedded quotes and hide reader-state bugs; hex literals also preserve
binary NULs in fixtures.
+ def rawRecords = [
+ "x| qa|bq|c", "abcqleft|rightq|tail", "qleft\nrightq|tail",
"left|qunclosed",
+ "qaqqbq,tail", "eeabc,tail", "a\u0000b,tail", "", "|", ",",
"qleftq|tail",
+ "x|\u2003\u2003qa|bq|c", "a\u0000\u0000b,tail", "a\\b,tail",
+ '"a""b",tail', '"a\\"b",tail', "abcqleftérightqétail"
+ ]
+ String rawExpressions = rawRecords.collect {
+ "decode(unhex('${it.getBytes('UTF-8').encodeHex()}'), 'UTF-8')"
+ }.join(", ")
+ // PostgreSQL-backed Hive metastores cannot persist NUL in text
properties. Disabled quote/escape
+ // settings remain covered by the Hive SerDe oracle and V2 reader unit
tests, without a metastore.
+ def dialects = [
+ [table: "csv_raw_custom", separator: "|", quote: "q", escape: "e"],
+ [table: "csv_raw_backslash", separator: ",", quote: "q", escape:
sqlBackslash],
+ [table: "csv_raw_default", separator: ",", quote: '"', escape:
sqlBackslash],
+ [table: "csv_raw_utf8", separator: "é", quote: "q", escape: "e"]
+ ]
+ for (def dialect : dialects) {
+ hive_docker "DROP TABLE IF EXISTS
csv_table_properties_db.${dialect.table}"
+ hive_docker """
+ CREATE TABLE csv_table_properties_db.${dialect.table} (raw_record
STRING) STORED AS TEXTFILE
+ """
+ hive_docker """
+ INSERT INTO csv_table_properties_db.${dialect.table}
+ SELECT explode(array(${rawExpressions}))
+ """
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.${dialect.table}
+ REPLACE COLUMNS (first_value STRING, second_value STRING,
third_value STRING)
+ """
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.${dialect.table}
+ SET SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
+ """
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.${dialect.table} SET
TBLPROPERTIES (
+ 'separatorChar'='${dialect.separator}',
'quoteChar'='${dialect.quote}',
+ 'escapeChar'='${dialect.escape}'
+ )
+ """
+ sql "REFRESH DATABASE
csv_table_properties_catalog.csv_table_properties_db"
+ for (String query : [
+ "SELECT coalesce(hex(first_value), 'NULL'),
coalesce(hex(second_value), 'NULL'), " +
+ "coalesce(hex(third_value), 'NULL') FROM %s ORDER BY 1, 2, 3",
+ "SELECT count(*), count(first_value), count(second_value),
count(third_value) FROM %s",
+ "SELECT coalesce(hex(third_value), 'NULL'), hex(first_value) FROM
%s " +
+ "WHERE first_value IS NOT NULL ORDER BY 1, 2"
+ ]) {
+ def expected = hive_docker(String.format(query,
"csv_table_properties_db.${dialect.table}"))
+ checkV2 {
+ assertEquals(expected, sql(String.format(query,
dialect.table)))
+ }
+ }
+ }
+
+ // These metadata values initialize successfully in Hive, but its reader
rejects the effective tuple.
+ for (String properties : [
+ "'separatorChar'='q', 'quoteChar'='q', 'escapeChar'='e'",
+ "'separatorChar'='e', 'quoteChar'='q', 'escapeChar'='e'",
+ "'separatorChar'='s', 'quoteChar'='e', 'escapeChar'='e'",
+ "'separatorChar'='${sqlBackslash}', 'quoteChar'='q', 'escapeChar'='\"'"
+ ]) {
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.csv_two_columns SET
TBLPROPERTIES (${properties})
+ """
+ sql "REFRESH DATABASE
csv_table_properties_catalog.csv_table_properties_db"
+ test {
+ sql "SELECT label, payload FROM csv_two_columns ORDER BY label"
+ exception "separatorChar, quoteChar and escapeChar must be
distinct when non-NUL"
+ }
+ }
+ hive_docker """
+ ALTER TABLE csv_table_properties_db.csv_two_columns SET TBLPROPERTIES (
+ 'separatorChar'='s', 'quoteChar'='q', 'escapeChar'='e'
+ )
+ """
+ sql "REFRESH DATABASE csv_table_properties_catalog.csv_table_properties_db"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
