morningman opened a new pull request, #68952:
URL: https://github.com/apache/doris/pull/68952
### What problem does this PR solve?
Issue Number: None
Related PR: #66334, #66547, #66573, #67395, #66399
Problem Summary:
**In short.** This ports the tests of three branch-4.1 PRs that master
already implements but does not pin: Paimon VARIANT reads through the JNI
reader (#66334), Variant sub-path access paths for that reader (#66547) and
nested type pruning for it (#66573). It changes no production code; besides
tests, it extends one docker fixture.
**Background.** In the Paimon JNI reader, a VARIANT column crosses JNI in
Paimon's encoded layout (metadata dictionaries, a metadata id per row, value
bytes), and the BE fills a Variant V2 column from it. With #66547 the BE passes
the sub-paths of each projected Variant column to the Java scanner as
`variant_access_path.<column>.<n>`. The scanner turns them into a Paimon read
type with metadata-marked Variant fields, so Paimon reads only those paths
(from the shredded columns when the file has them), and the scanner rebuilds a
partial Variant. #66573 does the same for ROW, ARRAY and MAP sub-fields. On
master this arrived with #67395 (Paimon Variant V2 and JNI reads) and #66399
(nested column pruning for Paimon through the `SUPPORTS_NESTED_COLUMN_PRUNE`
connector capability). Comparing each branch-4.x hunk with master, with
branch-4.2 as the final 4.x form, found the production behavior equivalent;
only the tests did not come along.
**The problem, and what it cost.** These behaviors are unpinned on master:
| Behavior | Tested on master before this PR |
|---|---|
| A nullable Variant column filled from Paimon's encoded rows: outer null
map, two metadata dictionaries with deduplicated ids, values decoded per row |
only one non-nullable row with a single empty dictionary |
| `variant_access_path.<column>.<n>` keyed by the projected column position,
which is how the Java scanner finds the paths of each required field | not on
the BE side, not on the Java side |
| OPTIONAL Paimon shredding wrappers accepted for an object field and for an
array element | only an OPTIONAL metadata field |
| The Variant JNI layout written by the Java side: null map, metadata bytes,
value bytes, metadata ids, value offsets | offsets and counts only |
| The scanner's Variant projection: metadata-marked read type, partial
object rebuild, JSON null kept and missing path omitted, whole-column fallback
for numeric, dotted or overlapping paths, and the guard against the misaligned
outer null bitmap of Paimon 1.4.2 | one case (every path missing) |
| The Paimon connector declares `SUPPORTS_NESTED_COLUMN_PRUNE` but not
`SUPPORTS_FIELD_ID_ACCESS_PATH` (with field-id paths every segment would be
`-1` and the pruned leaves would read back NULL) | no; the fe-core tests use a
mock connector |
| End to end: JNI projection of a shredded file, a nested shredded path, an
array-index path falling back to the whole column, mixed shredded and
unshredded files, and the access paths in EXPLAIN | no Paimon case |
**How this PR fixes it.**
- BE: `be/test/format/jni/jni_data_bridge_test.cpp` (new, as on branch-4.x);
`PaimonJniReaderTest.PublishesVariantAccessPathsByProjectedColumnPosition`;
`ParquetSchemaTest.AppliesPaimonShreddedVariantOverrideWithOptionalFields`,
which replaces the metadata-only case it extends.
- BE Java (JUnit 5): six assertions in `VectorColumnVariantTest`;
`PaimonColumnValueTest#testGetVariantEncodedBytes`;
`PaimonJniScannerTest#testVariantAccessPathsStayAlignedWithRequiredFields`;
four `PaimonVariantProjectionTest` cases.
- FE:
`PaimonConnectorMetadataMvccTest#connectorDeclaresNestedColumnPruneButNotFieldIdAccessPath`.
- Regression: `test_paimon_catalog_variant` gets the four JNI query blocks
and the three EXPLAIN checks of #66547. The `variant_shredded` fixture in
`run13.sql` gets the shredding schema of #66547 with `profile.city`, and the
`profile` and `tags` values two of the blocks read. The docker Spark runs
Paimon 1.3.1 (branch-4.x uses 1.4.2); 1.3.1 writes this schema, and the data
files hold `profile.city` as a shredded STRING column. No other suite reads
`variant_shredded`.
In the branch-4.x suite, one EXPLAIN block has both `contains` and `check`;
the regression framework skips `contains` once `check` is set, so that
access-path assertion never ran. Here the access-path checks are part of the
`check` closures, and the split checks use the regex form master already uses.
Not ported: the tests of the `enable_variant_v2` FE switch, which master
removed in #66858, and the blocks that need other branch-4.x PRs (a primary-key
Variant table, deeper shredded objects, the Variant row-group profile counters).
**Results.** All the tests above pass locally, and the regression suite
passes against the Iceberg docker environment with the new fixture; the 17
existing blocks of its `.out` are unchanged and the 4 new ones give the same
rows as branch-4.x. Each new unit test fails when the logic it guards is
changed, while the other tests of the same suites stay green:
| Change to the code under test | Test that fails |
|---|---|
| BE gives every Variant row metadata dictionary 0 |
`JniDataBridgeTest.FillNullableVariantV2FromEncodedPaimonLayout` |
| BE numbers `variant_access_path` keys among the Variant columns instead of
by projected position |
`PaimonJniReaderTest.PublishesVariantAccessPathsByProjectedColumnPosition` |
| BE accepts an OPTIONAL wrapper only for array elements |
`ParquetSchemaTest.AppliesPaimonShreddedVariantOverrideWithOptionalFields` |
| Java copies one byte less of each metadata dictionary |
`VectorColumnVariantTest.testVariantTypeAndEncodedLayout` |
| The projection trusts Paimon's outer Variant null bit |
`PaimonVariantProjectionTest.testIgnoresMisalignedPaimonVariantNullBitmap` |
| The scanner reads access path indexes from 1 |
`PaimonJniScannerTest.testVariantAccessPathsStayAlignedWithRequiredFields` |
| `getVariantMetadata` returns the value bytes |
`PaimonColumnValueTest.testGetVariantEncodedBytes` |
| The connector also declares `SUPPORTS_FIELD_ID_ACCESS_PATH` |
`PaimonConnectorMetadataMvccTest.connectorDeclaresNestedColumnPruneButNotFieldIdAccessPath`
|
Original author of the ported tests: @zhangstar333
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `test_paimon_catalog_variant` (`.out` regenerated by the script)
and `test_paimon_jni_complex_column_pruning` pass on a local cluster against
the Iceberg docker environment with the updated `run13.sql`.
- [x] Unit Test
- BE: `JniDataBridgeTest`, `PaimonJniReaderTest` and
`ParquetSchemaTest` (58 tests). Java: `VectorColumnVariantTest`,
`PaimonColumnValueTest`, `PaimonJniScannerTest`, `PaimonVariantProjectionTest`.
FE: `PaimonConnectorMetadataMvccTest`. All pass, and each new test fails under
the matching change in the table above.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
