morningman opened a new pull request, #69301:
URL: https://github.com/apache/doris/pull/69301
### What problem does this PR solve?
Issue Number: None
Related PR: #68883, #69194
Problem Summary:
**In short.** An INSERT of JSON with many distinct keys into a VARIANT
column of an Iceberg or Paimon table fails with an Arrow capacity error, or
runs out of memory first, because the writers copy the metadata dictionary of a
whole batch into every row. This PR writes a row whose shared dictionary is
much larger than its value with a dictionary of only its own keys, as the
Flight path already does, and keeps small shared dictionaries as they are.
**Background.** A Variant value is a metadata dictionary of keys plus a
value that refers to keys by their index. `parse_json()` encodes a block of
rows with one `JsonStringToVariantEncoder`, so all rows of the block share one
dictionary holding the keys of every row. Iceberg
(`write_column_to_iceberg_arrow`, which passes `compact_metadata = false`) and
Paimon (`write_binary_variant_arrow`) copy that dictionary into the metadata of
every row they write, and the Paimon writer validates it once per row. The
Flight path (#68883) instead re-encodes a row that does not use its whole
dictionary.
**The problem, and what it cost.** The bytes a batch writes into the
metadata column, and on Paimon the validation work, grow as rows × dictionary
bytes. 4064 rows that each have a distinct 248-byte key share a 1 MiB
dictionary, so the metadata column needs about 4 GiB, past the 2 GiB an Arrow
binary array holds. On a local cluster, `INSERT ... SELECT number,
PARSE_TO_VARIANT(<one such key per row>) FROM numbers("number" = "8192")`:
- into a Paimon table failed with `Variant V2 Arrow append failed: Capacity
error: array cannot contain more than 2147483646 bytes, have 2148527183`;
- into an Iceberg v3 table hit the same Arrow error, which
`checkArrowStatus` turns into a `FatalError`, so a debug or ASAN BE aborted (a
release BE fails the INSERT).
Nothing was written. Below that limit the copy still costs memory, and the
Paimon validation is quadratic in the number of keys. branch-4.1 and branch-4.2
have the same code (found by reading it).
**How this PR fixes it.** `with_row_metadata()` in
`data_type_variant_v2_serde.cpp` decides the dictionary of each row, under one
of two policies:
- `OWN_KEYS` (Flight, unchanged): a row that does not use its whole
dictionary is re-encoded with a dictionary of only its keys.
- `BOUNDED` (Iceberg and Paimon): a row keeps the dictionary it shares with
its batch unless that is more than 1 KiB larger than the row's value; then it
is re-encoded as under `OWN_KEYS`.
So a table write copies at most the value bytes plus 1 KiB of metadata a row
(or the row's own keys), and re-encoding a row costs no more than copying the
dictionary would have. A small dictionary stays shared, so the rows keep
identical metadata that Parquet dictionary-encodes, and nothing changes for
them. The Paimon writer now checks the size limits and validates the dictionary
of the row as written, so a re-encoded row is validated against its own keys.
Re-encoding changes the key indexes in the value, not what the value means;
readers decode each row with its own metadata.
**Results.**
| Batch | Before | After |
|---|---|---|
| 256 rows, a distinct 200-byte key each (BE UT) | each row carries the 52
KiB dictionary (13 MiB) | each row carries its own key |
| A row whose 64 KiB value outweighs that dictionary | shared dictionary |
shared dictionary (unchanged) |
| Small shared dictionary | shared dictionary | shared dictionary, bytes
unchanged |
| INSERT of 8192 rows with distinct 248-byte keys into Iceberg v3 / Paimon
(regression) | Iceberg: Arrow capacity error (an ASAN BE aborts); Paimon: Arrow
capacity error | both succeed; Doris and Spark read every row back as written |
### Release note
Fix INSERT into Iceberg and Paimon VARIANT columns failing with an Arrow
capacity error, or running out of memory, when a batch of JSON has many
distinct keys.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- New `test_iceberg_write_variant_many_keys` and
`test_paimon_write_variant_many_keys`: the INSERT above, then Doris checks
every row against the JSON it wrote and Spark (`to_json`) does the same. They
pass on a local cluster with the iceberg docker environment; on a BE built with
master's serde the INSERTs fail as described. The existing
`test_iceberg_variant_read`, `test_iceberg_variant_defaults` and the six
`test_paimon_write_variant*` suites pass.
- [x] Unit Test
- `DataTypeVariantV2SerdeOutputTest`:
`TableWritersGiveRowsTheirOwnKeysInsteadOfALargeDictionary` (Iceberg and Paimon
layouts; every row decodes to its key and value) and
`TableWritersKeepASmallSharedDictionary`. These and the related Variant serde,
Paimon convertor, Flight Variant, Arrow and Parquet writer tests (32) pass.
Each of these mutations turns the new tests red: Iceberg compacting every row,
keeping every shared dictionary, Paimon not compacting, an absolute 1 KiB
threshold instead of one relative to the value, and treating table writers as
Flight.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
