This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new bdcd7eacc79 [improvement](be) Reuse memory-tracked scratch when
validating Variant payloads (#69019)
bdcd7eacc79 is described below
commit bdcd7eacc790bfa5fbb7fb13d4849d1a610579a2
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Oct 15 14:49:24 2026 +0800
[improvement](be) Reuse memory-tracked scratch when validating Variant
payloads (#69019)
### What problem does this PR solve?
Issue Number: None
Related PR: #68959
Problem Summary:
**In short.** Validating a Variant value no longer allocates a fresh,
untracked `std::vector` for every object it checks. A new
`VariantPayloadValidator` keeps that scratch in one `DorisVector` that
every later row reuses, and the callers that validate whole batches
(encoded-row inserts, the encoded-row appender that #68959 uses to
import unshredded Parquet Variant values, the native Arrow Variant
reader and `sanity_check`) now hold one validator per batch, or for the
appender's whole life. This was raised in the review of #68959.
**Background.** Encoded Variant bytes that come from outside (Parquet
files, JNI readers, Arrow batches) are validated before a
`ColumnVariantV2` takes them. For an object,
`validate_variant_payload()` checks that the field values exactly tile
the object's value section: it collects the byte span of every field
value, sorts the spans by offset and rejects overlaps, gaps and trailing
bytes. `require_valid_object()` collected those spans into a local
`std::vector` with `reserve(field_count)`, so every object it checked,
nested objects included, allocated once and freed the vector when its
check returned. `VariantBatchBuilder`, which the Parquet importer used
before #68959, checked the same intervals in a `DorisVector` it reuses
across rows.
**The problem, and what it cost.**
| Validating a batch of 4096 object rows | master | with this PR |
|---|---|---|
| Scratch allocations | at least one per object, so at least 4096 per
batch, plus one per nested object | one buffer per batch (one per
appender for the Parquet importer); it grows to the widest path of
objects and is then reused |
| Allocator of the scratch (16 bytes per field) | plain `malloc` through
`std::allocator`, which the Doris memory trackers do not see | the Doris
`Allocator`, counted by the thread's memory tracker |
`insert_encoded_rows()` (JNI readers, inserted encoded values, Variant
element extraction), the encoded-row appender and the native Arrow
Variant reader validate every row, so this is per-row work on the read
path.
**How this PR fixes it.**
- `VariantPayloadValidator` (new header
`core/value/variant/variant_payload_validator.h`) keeps the spans in one
`DorisVector` used as a stack. An object pushes the span of each field
after checking that field, so a nested object pushes its spans above
those of its ancestors, sorts and checks only its own range and pops
back before returning. The checks, their order and their error messages
are unchanged. A call that throws leaves the entries of its unfinished
objects behind, so `validate()` starts by clearing the stack.
- The class is defined in `variant_field.cpp` next to the checks it
shares with `validate_variant_payload()`, which remains the one-shot
form for a single value (`VariantField::from_ref` / `from_bytes`,
`parse_canonical_serialized`). It is declared in its own header so that
`core/field.h`, which includes `variant_field.h`, does not pull in
`core/custom_allocator.h` (its include-closure budget has no slack).
- `ColumnVariantV2::insert_encoded_rows()`,
`ColumnVariantV2::sanity_check()` and the native Arrow reader in
`DataTypeVariantV2SerDe` create one validator per call;
`EncodedRowsAppender` keeps one in its `Impl`, so all the chunks a
Parquet reader imports share it.
**Results.**
- The reusable validator accepts and rejects the same 13 payloads as the
one-shot call: nested and wide objects, objects inside arrays, and the
corrupt objects of the existing tests (duplicate offset, overlapping
spans, gap, trailing bytes, unordered keys). A corrupt object nested in
field `b` throws after three entries are on the stack, and the next
valid payload still validates
(`ReusableValidatorMatchesOneShotValidation`).
- Validating a 1000-field object and a nested object 4096 times each
leaves the scratch capacity where the first row put it
(`ReusableValidatorReusesScratchAcrossRows`); the appender keeps the
same capacity across two 4096-row chunks
(`EncodedRowsAppenderReusesValidationScratchAcrossChunks`).
- The scratch of a 4096-field object shows up in the thread's memory
tracker (at least 64 KiB while the validator lives) and is released with
it (`ReusableValidatorScratchIsMemTrackerVisible`).
- Each new test catches the regression it is written for: with the
appender back on the one-shot call, only
`EncodedRowsAppenderReusesValidationScratchAcrossChunks` fails; with the
scratch back on `std::vector`, only
`ReusableValidatorScratchIsMemTrackerVisible` fails; without the
`clear()` at the start of `validate()`,
`ReusableValidatorMatchesOneShotValidation` stops at the `DCHECK` that
the stack is empty after a valid payload.
- The 233 tests of the Variant suites listed below pass locally, except
`ColumnVariantV2Test.TypedComparisonBoundariesMatchEncodedWithoutMaterializing`,
which also fails on master on the macOS machine used here and passes in
CI.
---
.../core/column/variant_v2/column_variant_v2.cpp | 19 ++-
be/src/core/column/variant_v2/column_variant_v2.h | 1 +
.../data_type_serde/data_type_variant_v2_serde.cpp | 4 +-
be/src/core/value/variant/variant_field.cpp | 49 ++++---
.../core/value/variant/variant_payload_validator.h | 67 +++++++++
be/test/core/column/column_variant_v2_test.cpp | 28 ++++
be/test/util/variant/variant_field_test.cpp | 149 +++++++++++++++++++++
7 files changed, 295 insertions(+), 22 deletions(-)
diff --git a/be/src/core/column/variant_v2/column_variant_v2.cpp
b/be/src/core/column/variant_v2/column_variant_v2.cpp
index edab4bc6ecb..65a0d9e7cb6 100644
--- a/be/src/core/column/variant_v2/column_variant_v2.cpp
+++ b/be/src/core/column/variant_v2/column_variant_v2.cpp
@@ -45,6 +45,7 @@
#include "core/value/variant/variant_canonical.h"
#include "core/value/variant/variant_field.h"
#include "core/value/variant/variant_parquet_encoding.h"
+#include "core/value/variant/variant_payload_validator.h"
#include "exec/sort/sort_block.h"
namespace doris {
@@ -876,12 +877,13 @@ void ColumnVariantV2::sanity_check() const {
const StringRef metadata = metadatas.get_data_at(id);
validate_variant_metadata({metadata.data, metadata.size});
}
+ VariantPayloadValidator payload_validator;
for (size_t row = 0; row < values.size(); ++row) {
const uint32_t id = metadata_ids[row];
DORIS_CHECK_LT(id, metadatas.size()) << "ColumnVariantV2 metadata
id is out of range";
const StringRef metadata = metadatas.get_data_at(id);
- validate_variant_payload({.metadata = {.data = metadata.data,
.size = metadata.size},
- .value = values.get_data_at(row)});
+ payload_validator.validate({.metadata = {.data = metadata.data,
.size = metadata.size},
+ .value = values.get_data_at(row)});
}
}
}
@@ -973,12 +975,13 @@ void ColumnVariantV2::insert_encoded_rows( //
NOLINT(readability-function-size)
// Validate every row before changing a typed destination or appending to
an encoded one.
// This keeps input errors atomic without introducing a second temporary
column.
+ VariantPayloadValidator payload_validator;
if (data.meta_ids.empty()) {
const VariantMetadataRef metadata = metadata_at(0);
for (size_t row = 0; row < rows; ++row) {
const uint32_t begin = data.value_offsets[row];
const uint32_t end = data.value_offsets[row + 1];
- validate_variant_payload(
+ payload_validator.validate(
{.metadata = metadata,
.value = {data.value_bytes.data + begin,
static_cast<size_t>(end - begin)}});
}
@@ -988,7 +991,7 @@ void ColumnVariantV2::insert_encoded_rows( //
NOLINT(readability-function-size)
DORIS_CHECK_LT(source_id, metadata_count) << "encoded row metadata
id is out of range";
const uint32_t begin = data.value_offsets[row];
const uint32_t end = data.value_offsets[row + 1];
- validate_variant_payload(
+ payload_validator.validate(
{.metadata = metadata_at(source_id),
.value = {data.value_bytes.data + begin,
static_cast<size_t>(end - begin)}});
}
@@ -1007,6 +1010,8 @@ struct ColumnVariantV2::EncodedRowsAppender::Impl {
ColumnVariantV2* column;
MetadataHashHeads metadata_hash_heads;
DorisVector<uint32_t> metadata_hash_next;
+ // Kept across append() calls so that every chunk a reader imports reuses
one scratch.
+ VariantPayloadValidator payload_validator;
size_t expected_rows = 0;
bool initialized = false;
#ifdef BE_TEST
@@ -1025,6 +1030,10 @@
ColumnVariantV2::EncodedRowsAppender::EncodedRowsAppender(EncodedRowsAppender&&)
size_t ColumnVariantV2::EncodedRowsAppender::metadata_comparisons_for_test()
const noexcept {
return _impl == nullptr ? 0 : _impl->metadata_comparisons;
}
+
+size_t
ColumnVariantV2::EncodedRowsAppender::validation_scratch_capacity_for_test()
const noexcept {
+ return _impl == nullptr ? 0 :
_impl->payload_validator.object_span_capacity_for_test();
+}
#endif
void ColumnVariantV2::EncodedRowsAppender::append( //
NOLINT(readability-function-size)
@@ -1094,7 +1103,7 @@ void ColumnVariantV2::EncodedRowsAppender::append( //
NOLINT(readability-functio
if (!source_metadata_ids.empty()) {
source_metadata_ids[row] = source_metadata_id;
}
- validate_variant_payload(value);
+ _impl->payload_validator.validate(value);
source_values[row] = value.value;
if (value.value.size > std::numeric_limits<size_t>::max() -
total_value_bytes) {
throw Exception(ErrorCode::INVALID_ARGUMENT,
diff --git a/be/src/core/column/variant_v2/column_variant_v2.h
b/be/src/core/column/variant_v2/column_variant_v2.h
index 976a4a66e31..e54b7ad3e05 100644
--- a/be/src/core/column/variant_v2/column_variant_v2.h
+++ b/be/src/core/column/variant_v2/column_variant_v2.h
@@ -137,6 +137,7 @@ public:
#ifdef BE_TEST
size_t metadata_comparisons_for_test() const noexcept;
+ size_t validation_scratch_capacity_for_test() const noexcept;
#endif
private:
diff --git a/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp
b/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp
index 86c9b6c2c1f..6c183a8eb68 100644
--- a/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_variant_v2_serde.cpp
@@ -50,6 +50,7 @@
#include "core/value/jsonb_value.h"
#include "core/value/variant/variant_batch_builder.h"
#include "core/value/variant/variant_field.h"
+#include "core/value/variant/variant_payload_validator.h"
#include "exprs/function/parse/variant_jsonb_parse.h"
#include "exprs/function/parse/variant_string_parse.h"
#include "util/jsonb_document.h"
@@ -541,6 +542,7 @@ Status
DataTypeVariantV2SerDe::read_column_from_arrow(IColumn& column, const arr
// A driver supplies untrusted bytes. Validate into a temporary batch
before mutating
// the destination, and copy buffers so the column survives Arrow
batch release.
VariantBatchBuilder decoded;
+ VariantPayloadValidator payload_validator;
for (int64_t i = 0; i < slice->length(); ++i) {
auto row = decoded.begin_row();
if (storage.IsNull(i)) {
@@ -556,7 +558,7 @@ Status
DataTypeVariantV2SerDe::read_column_from_arrow(IColumn& column, const arr
(VariantRef {.metadata = {.data = meta.data(), .size =
meta.size()},
.value = {value.data(), value.size()}});
validate_variant_metadata(ref.metadata);
- validate_variant_payload(ref);
+ payload_validator.validate(ref);
row.add_value(ref);
}
row.finish();
diff --git a/be/src/core/value/variant/variant_field.cpp
b/be/src/core/value/variant/variant_field.cpp
index ff5354e64b5..620995a3731 100644
--- a/be/src/core/value/variant/variant_field.cpp
+++ b/be/src/core/value/variant/variant_field.cpp
@@ -20,11 +20,12 @@
#include <algorithm>
#include <cstring>
#include <limits>
+#include <span>
#include <utility>
-#include <vector>
#include "common/exception.h"
#include "core/value/variant/variant_parquet_encoding.h"
+#include "core/value/variant/variant_payload_validator.h"
#include "core/value/variant/variant_scalar.h"
#include "util/utf8_check.h"
@@ -128,13 +129,6 @@ void require_valid_primitive(VariantRef value) {
throw Exception(ErrorCode::CORRUPTION, "VariantField contains an unknown
primitive type");
}
-void require_exact_value(VariantRef value, uint32_t depth);
-
-struct ObjectValueSpan {
- size_t offset;
- size_t size;
-};
-
size_t object_values_offset(VariantRef value, uint32_t count) {
const uint8_t value_header =
static_cast<uint8_t>(value.value.data[0]) >>
VARIANT_VALUE_HEADER_SHIFT;
@@ -148,13 +142,24 @@ size_t object_values_offset(VariantRef value, uint32_t
count) {
(static_cast<size_t>(count) + 1) * offset_width;
}
-void require_valid_object(VariantRef value, uint32_t depth) {
+} // namespace
+
+void VariantPayloadValidator::require_valid_object(VariantRef value, uint32_t
depth) {
const uint32_t count = value.num_elements();
const size_t values_offset = object_values_offset(value, count);
const char* values_begin = value.value.data + values_offset;
const size_t values_size = value.value.size - values_offset;
- std::vector<ObjectValueSpan> spans;
- spans.reserve(count);
+ // Entries below base belong to the ancestors of this object; see
_object_spans.
+ const size_t base = _object_spans.size();
+ // Reserve this object's spans up front. A fresh validator, such as the
one each
+ // validate_variant_payload() call makes, then allocates once for a flat
object of 1000
+ // fields instead of growing ten times (1, 2, 4, ..., 1024). Reserving
exactly base + count
+ // would not be enough for nested objects: in a root of 26 fields that are
each objects of
+ // 30 fields, field k needs k + 30 entries, so every later sibling would
reallocate by one
+ // entry. Taking at least twice the capacity grows 26 -> 52 -> 104 instead.
+ if (_object_spans.capacity() < base + count) {
+ _object_spans.reserve(std::max(base + count, 2 *
_object_spans.capacity()));
+ }
StringRef previous_key;
for (uint32_t index = 0; index < count; ++index) {
@@ -166,11 +171,12 @@ void require_valid_object(VariantRef value, uint32_t
depth) {
"VariantField object keys are not strictly ordered
at field {}", index);
}
require_exact_value(child, depth + 1);
- spans.push_back({.offset = static_cast<size_t>(child.value.data -
values_begin),
- .size = child.value.size});
+ _object_spans.push_back({.offset =
static_cast<size_t>(child.value.data - values_begin),
+ .size = child.value.size});
previous_key = key;
}
+ const std::span<ObjectValueSpan> spans(_object_spans.data() + base, count);
std::ranges::sort(spans, [](const ObjectValueSpan& left, const
ObjectValueSpan& right) {
return left.offset < right.offset;
});
@@ -190,16 +196,17 @@ void require_valid_object(VariantRef value, uint32_t
depth) {
throw Exception(ErrorCode::CORRUPTION,
"VariantField object values contain unreferenced
trailing bytes");
}
+ _object_spans.resize(base);
}
-void require_valid_array(VariantRef value, uint32_t depth) {
+void VariantPayloadValidator::require_valid_array(VariantRef value, uint32_t
depth) {
const uint32_t count = value.num_elements();
for (uint32_t index = 0; index < count; ++index) {
require_exact_value(value.array_at(index), depth + 1);
}
}
-void require_exact_value(VariantRef value, uint32_t depth) {
+void VariantPayloadValidator::require_exact_value(VariantRef value, uint32_t
depth) {
if (depth > VARIANT_MAX_NESTING_DEPTH) {
throw Exception(ErrorCode::CORRUPTION,
"VariantField value exceeds maximum nesting depth {}",
@@ -229,6 +236,8 @@ void require_exact_value(VariantRef value, uint32_t depth) {
throw Exception(ErrorCode::CORRUPTION, "VariantField contains an unknown
value type");
}
+namespace {
+
struct RowSlices {
VariantMetadataRef metadata;
VariantRef value;
@@ -276,9 +285,17 @@ void validate_variant_metadata(VariantMetadataRef
metadata) {
}
}
-void validate_variant_payload(VariantRef value) {
+void VariantPayloadValidator::validate(VariantRef value) {
require_non_null(value.value, "value");
+ // A call that threw in the middle of an object left the spans of its
unfinished objects
+ // behind, and the next call must not count them as its own.
+ _object_spans.clear();
require_exact_value(value, 0);
+ DCHECK(_object_spans.empty());
+}
+
+void validate_variant_payload(VariantRef value) {
+ VariantPayloadValidator().validate(value);
}
VariantField::VariantField(std::unique_ptr<char[]> data, size_t size) noexcept
diff --git a/be/src/core/value/variant/variant_payload_validator.h
b/be/src/core/value/variant/variant_payload_validator.h
new file mode 100644
index 00000000000..959b4198850
--- /dev/null
+++ b/be/src/core/value/variant/variant_payload_validator.h
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstddef>
+#include <cstdint>
+
+#include "core/custom_allocator.h"
+#include "core/value/variant/variant_value.h"
+
+namespace doris {
+
+// Reusable form of validate_variant_payload() (variant_field.h), which is the
one-shot call for a
+// single value. Checking an object needs the byte spans of its field values,
and this validator
+// keeps them in one memory-tracked buffer that every later call reuses. A
caller that validates
+// many rows holds one validator for all of them, so a batch of 4096 object
rows grows the buffer a
+// few times instead of allocating once per object. The class is defined in
variant_field.cpp next
+// to the checks it shares with validate_variant_payload(); it is declared
apart from
+// variant_field.h so that core/field.h, which includes that header, does not
also pull in
+// core/custom_allocator.h.
+class VariantPayloadValidator {
+public:
+ // Validate exactly one recursive Variant payload, with the same checks
and errors as
+ // validate_variant_payload(). The referenced metadata must already have
passed
+ // validate_variant_metadata().
+ void validate(VariantRef value);
+
+#ifdef BE_TEST
+ size_t object_span_capacity_for_test() const noexcept { return
_object_spans.capacity(); }
+#endif
+
+private:
+ // The bytes one field value takes inside its object's value section.
+ struct ObjectValueSpan {
+ size_t offset;
+ size_t size;
+ };
+
+ void require_exact_value(VariantRef value, uint32_t depth);
+ void require_valid_object(VariantRef value, uint32_t depth);
+ void require_valid_array(VariantRef value, uint32_t depth);
+
+ // A stack shared by the objects on the path from the root to the value
being checked. Each
+ // object pushes the span of a field after checking that field, so a
nested object pushes its
+ // own spans above those of its ancestors and pops them before it returns.
For
+ // {"a":1,"b":{"c":2,"d":3},"e":4}, the stack holds [a] when the root
starts checking "b";
+ // "b" grows it to [a, c, d], checks its own two entries and pops back to
[a], and the root
+ // then pushes "b" and "e" and checks [a, b, e].
+ DorisVector<ObjectValueSpan> _object_spans;
+};
+
+} // namespace doris
diff --git a/be/test/core/column/column_variant_v2_test.cpp
b/be/test/core/column/column_variant_v2_test.cpp
index aa645e471ad..0a20eafca37 100644
--- a/be/test/core/column/column_variant_v2_test.cpp
+++ b/be/test/core/column/column_variant_v2_test.cpp
@@ -1226,6 +1226,34 @@ TEST(ColumnVariantV2Test,
EncodedRowsAppenderInternsHighCardinalityMetadataAcros
column->sanity_check();
}
+TEST(ColumnVariantV2Test,
EncodedRowsAppenderReusesValidationScratchAcrossChunks) {
+ constexpr size_t FIELDS = 64;
+ constexpr size_t ROWS = 4096;
+ std::string json = R"({"nested":{"x":1,"y":[{"z":2}]})";
+ for (size_t field = 0; field < FIELDS; ++field) {
+ json += ",\"k" + std::to_string(field) + "\":" + std::to_string(field);
+ }
+ json.push_back('}');
+ const VariantField wide = encode_json(json);
+ const std::vector<VariantRef> rows(ROWS, wide.ref());
+
+ auto column = ColumnVariantV2::create();
+ auto appender = column->create_encoded_rows_appender();
+ EXPECT_EQ(appender.validation_scratch_capacity_for_test(), 0);
+ appender.append(rows);
+ const size_t capacity = appender.validation_scratch_capacity_for_test();
+ EXPECT_GE(capacity, FIELDS + 1);
+ appender.append(rows);
+ EXPECT_EQ(appender.validation_scratch_capacity_for_test(), capacity);
+
+ ASSERT_EQ(column->size(), 2 * ROWS);
+ JsonWriter expected;
+ to_json(wide.ref(), expected);
+ EXPECT_EQ(json_at(*column, 0), expected.value);
+ EXPECT_EQ(json_at(*column, 2 * ROWS - 1), expected.value);
+ column->sanity_check();
+}
+
TEST(ColumnVariantV2Test, InvalidEncodedRowsDoNotPartiallyAppendEncodedState) {
const OwnedEncodedData invalid = late_invalid_encoded_rows();
for (const bool omit_single_metadata_ids : {false, true}) {
diff --git a/be/test/util/variant/variant_field_test.cpp
b/be/test/util/variant/variant_field_test.cpp
index 1c072fd80db..9e932f784f2 100644
--- a/be/test/util/variant/variant_field_test.cpp
+++ b/be/test/util/variant/variant_field_test.cpp
@@ -29,7 +29,10 @@
#include "common/exception.h"
#include "core/value/variant/variant_parquet_encoding.h"
+#include "core/value/variant/variant_payload_validator.h"
#include "exprs/function/parse/variant_string_parse.h"
+#include "runtime/memory/mem_tracker_limiter.h"
+#include "runtime/thread_context.h"
namespace doris {
namespace {
@@ -218,6 +221,36 @@ VariantField encode_json(std::string_view json) {
return VariantField::from_ref(block.value_at(0));
}
+std::string wide_object_json(size_t fields) {
+ std::string json = "{";
+ for (size_t field = 0; field < fields; ++field) {
+ if (field != 0) {
+ json.push_back(',');
+ }
+ json += "\"k" + std::to_string(field) + "\":" + std::to_string(field);
+ }
+ json.push_back('}');
+ return json;
+}
+
+bool one_shot_accepts(VariantRef value) {
+ try {
+ validate_variant_payload(value);
+ return true;
+ } catch (const Exception&) {
+ return false;
+ }
+}
+
+bool validator_accepts(VariantPayloadValidator& validator, VariantRef value) {
+ try {
+ validator.validate(value);
+ return true;
+ } catch (const Exception&) {
+ return false;
+ }
+}
+
// NOLINTNEXTLINE(readability-function-cognitive-complexity) -- GTest
assertions are the matrix.
TEST(VariantFieldTest, ScalarObjectAndArrayRoundTrip) {
const std::string empty_metadata = metadata({}, true);
@@ -562,5 +595,121 @@ TEST(VariantFieldTest, EncodeJsonStableSubset) {
EXPECT_EQ(encode_json("[]").ref().num_elements(), 0);
}
+TEST(VariantFieldTest, ReusableValidatorMatchesOneShotValidation) {
+ const VariantField nested =
+
encode_json(R"({"a":1,"b":{"c":2,"d":{"e":[{"f":3},{"g":{"h":4}}]}},"i":5})");
+ const VariantField wide = encode_json(wide_object_json(300));
+ const VariantField objects_in_array =
encode_json(R"([{"a":1},{"b":{"c":2}},{}])");
+ const VariantField scalar = encode_json("7");
+
+ const std::string metadata_bytes = metadata({"a", "b"}, true);
+ const std::string reversed_metadata = metadata({"b", "a"}, false);
+ const std::string null_value = primitive(VariantPrimitiveId::NULL_VALUE);
+ const std::string true_value = primitive(VariantPrimitiveId::TRUE_VALUE);
+ const std::string two_values = object({0, 1}, {0, 1}, {null_value,
true_value});
+ const std::string duplicate_offset = object({0, 1}, {0, 0}, {null_value,
true_value});
+ const std::string overlapping_spans = object({0, 1}, {0, 1},
{long_string("x")});
+ const std::string internal_gap = object({0, 1}, {0, 2}, {null_value,
null_value, true_value});
+ const std::string trailing_unreferenced =
+ object({0, 1}, {0, 1}, {null_value, true_value, null_value});
+ // Only the object in field "b" is corrupt. It throws after the root has
pushed the span of
+ // "a" and it has pushed two spans of its own; the next validate() must
not inherit them.
+ const std::string gap_in_second_field = object({0, 1}, {0, 1},
{null_value, internal_gap});
+
+ struct Case {
+ VariantRef value;
+ bool valid;
+ };
+ const std::vector<Case> cases {
+ {.value = nested.ref(), .valid = true},
+ {.value = value_ref(metadata_bytes, gap_in_second_field), .valid =
false},
+ {.value = nested.ref(), .valid = true},
+ {.value = wide.ref(), .valid = true},
+ {.value = value_ref(metadata_bytes, duplicate_offset), .valid =
false},
+ {.value = objects_in_array.ref(), .valid = true},
+ {.value = value_ref(metadata_bytes, overlapping_spans), .valid =
false},
+ {.value = value_ref(metadata_bytes, internal_gap), .valid = false},
+ {.value = value_ref(metadata_bytes, trailing_unreferenced), .valid
= false},
+ {.value = value_ref(reversed_metadata, two_values), .valid =
false},
+ {.value = value_ref(metadata_bytes, two_values), .valid = true},
+ {.value = scalar.ref(), .valid = true},
+ {.value = wide.ref(), .valid = true},
+ };
+
+ VariantPayloadValidator validator;
+ for (size_t index = 0; index < cases.size(); ++index) {
+ EXPECT_EQ(one_shot_accepts(cases[index].value), cases[index].valid) <<
"case " << index;
+ EXPECT_EQ(validator_accepts(validator, cases[index].value),
cases[index].valid)
+ << "case " << index;
+ }
+}
+
+TEST(VariantFieldTest, ReusableValidatorReusesScratchAcrossRows) {
+ constexpr size_t FIELDS = 1'000;
+ constexpr size_t ROWS = 4'096;
+ const VariantField wide = encode_json(wide_object_json(FIELDS));
+ const VariantField nested =
encode_json(R"({"a":{"b":{"c":1,"d":2},"e":3},"f":[{"g":4}]})");
+
+ VariantPayloadValidator validator;
+ EXPECT_EQ(validator.object_span_capacity_for_test(), 0);
+ validator.validate(wide.ref());
+ const size_t capacity = validator.object_span_capacity_for_test();
+ EXPECT_GE(capacity, FIELDS);
+ for (size_t row = 0; row < ROWS; ++row) {
+ validator.validate(wide.ref());
+ validator.validate(nested.ref());
+ }
+ EXPECT_EQ(validator.object_span_capacity_for_test(), capacity);
+}
+
+TEST(VariantFieldTest, FreshValidatorReservesAnObjectOnce) {
+ // validate_variant_payload() and single-row callers such as
ColumnVariantV2::insert_data()
+ // validate with a fresh validator, so its first object must not grow from
zero.
+ constexpr size_t FIELDS = 1'000;
+ const VariantField wide = encode_json(wide_object_json(FIELDS));
+ VariantPayloadValidator flat_validator;
+ flat_validator.validate(wide.ref());
+ EXPECT_EQ(flat_validator.object_span_capacity_for_test(), FIELDS);
+
+ // 26 objects of 30 fields: the root reserves 26 entries, and its
children, which need
+ // index + 30, double the buffer twice instead of reallocating for every
sibling.
+ std::string nested_json = "{";
+ for (char key = 'a'; key <= 'z'; ++key) {
+ if (key != 'a') {
+ nested_json.push_back(',');
+ }
+ nested_json += '"';
+ nested_json += key;
+ nested_json += "\":" + wide_object_json(30);
+ }
+ nested_json.push_back('}');
+ const VariantField nested = encode_json(nested_json);
+ VariantPayloadValidator nested_validator;
+ nested_validator.validate(nested.ref());
+ EXPECT_EQ(nested_validator.object_span_capacity_for_test(), 104);
+}
+
+TEST(VariantFieldTest, ReusableValidatorScratchIsMemTrackerVisible) {
+ constexpr size_t FIELDS = 4'096;
+ // Each field of the object being checked takes one span of two size_t
values.
+ constexpr auto MIN_SCRATCH_BYTES = static_cast<int64_t>(FIELDS * 2 *
sizeof(size_t));
+ const VariantField wide = encode_json(wide_object_json(FIELDS));
+ const auto tracker =
MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::OTHER,
+
"VariantPayloadValidatorScratch");
+ auto scoped_tracker = SwitchThreadMemTrackerLimiter(tracker);
+ thread_context()->thread_mem_tracker_mgr->flush_untracked_mem();
+ const int64_t baseline = tracker->consumption();
+
+ {
+ VariantPayloadValidator validator;
+ validator.validate(wide.ref());
+ thread_context()->thread_mem_tracker_mgr->flush_untracked_mem();
+ EXPECT_GE(tracker->consumption(), baseline + MIN_SCRATCH_BYTES);
+ }
+
+ thread_context()->thread_mem_tracker_mgr->flush_untracked_mem();
+ EXPECT_EQ(tracker->consumption(), baseline);
+}
+
} // namespace
} // namespace doris
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
