yy782 opened a new pull request, #69276:
URL: https://github.com/apache/doris/pull/69276
### What problem does this PR solve?
Issue Number: #66492
Related PR: None
Problem Summary:
Changes are grouped into three areas:
**1. URI refactor**
**1.1 Replace the single-dataset members with one unified multi-dataset
representation**
- Removed from `ExternalFileTableValuedFunction`:
- `private long lanceDatasetVersion = -1;`
- `private List<LanceFragmentInfo> lanceFragments =
Collections.emptyList();`
- Replaced by a single field:
- `private List<LancePlannedDataset> lancePlannedDatasets` — each
`LancePlannedDataset`
carries `uri + version + fragments`.
- Why: the old two fields could only describe *one* dataset (one version,
one fragment
list), so they could not represent the multiple datasets that `uris`
introduces. With a
`List<LancePlannedDataset>`, a single `uri` is just a list of size 1 and
`uris` a list of
size N — both share the same in-memory shape. Downstream (`TVFScanNode`,
`LanceSplitBuilder`) no longer branches on uri vs uris; it only consumes
this list.
**1.2 New / changed APIs to enforce the `uri`/`uris` mutual exclusion**
- `StorageProperties` now declares `requireMutuallyExclusive(...)` — an
abstract method each
storage implements for its own single-location key, plus a static helper —
which rejects a
storage's location key together with `uris`.
- Every subclass (`AbstractS3CompatibleProperties`, `AzureProperties`,
`BrokerProperties`,
`HdfsCompatibleProperties`, `HdfsProperties`, `HttpProperties`,
`LocalProperties`,
`OSSHdfsProperties`) implements the check for its own location key.
- The TVF entry point is unified behind `getLancePlannedDatasets()`
(populated by
`setLancePlannedDatasets(datasets, referenceSchema)`);
`S3TableValuedFunction` /
`FileTableValuedFunction` / `TVFScanNode` consume it instead of the old
single-dataset
accessors.
- Why: the `uri`/`uris` mutual exclusion needs a single, enforced contract.
Putting
`requireMutuallyExclusive` on the `StorageProperties` base guarantees
every storage type
implements it, and keeps the check decoupled from each storage's concrete
location key — a
necessary API change to support `uris`.
**2. `uris` support (new feature)**
- New `Uris`: parses the `uris` JSON array into an ordered, de-duplicated URI
list; `MAX_URIS = 10000`; rejects non-array / empty / over-limit /
non-string
/ blank elements with `AnalysisException`.
- New `LanceMultiSnapshotLoader`: opens every dataset named by `uris`, pins
one
snapshot (version/schema/fragments) per dataset, returns them in array
order
(first is the reference); any open failure fails the whole query and names
the
path and operation; `MAX_DATASETS = 500`.
- New `LancePlannedDataset` (immutable uri+version+fragments) and
`LanceSchemaEquality` (compares schemas across datasets, used to pick the
reference schema).
- `LanceSplitBuilder.buildFragmentSplits(List<LancePlannedDataset>, ...)`:
builds
splits per dataset at its pinned version and normalizes scheduling weights
across all datasets in the plan.
- BE `lance_reader.{cpp,h}`: a split now carries its own dataset uri +
version
and opens the matching dataset.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. New `uris` TVF parameter (previously only `uri`).
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
