hsiang-c opened a new pull request, #9335:
URL: https://github.com/apache/iceberg/pull/9335
### Note to reviewers
- Closes https://github.com/apache/iceberg/issues/8856
- Instead of returning `ManifestFile` in
`BaseAllMetadataTableScan::reachableManifests`, we return a `Pair<Snapshot,
ManifestFile>` from all snapshots.
- `REF_SNAPSHOT_ID` is used by `AllManifestsTable` already, so I chose this
terminology instead of `AS_OF_SNAPSHOT`. I am open to other names as well.
- There are a few unit tests in Spark/Flink where the order of the return
values were not guaranteed. Therefore, I sorted the expected/actual values by
`file_path`.
- In the original PR, Russell commented
> These would allow us to be able to analyze the actual history in
all_entires and all_manifests
I also used `Pair<Snapshot, ManifestFile>` for all_* tables other than
`all_entries` and `all_manifests` but I'm not sure if it is necessary. Please
let me know.
### Sample output
- Insert 4 times, delete all of them, then insert one more time.
- Output is ordered by `reference_snapshot_timestamp_millis`
```
+-------------------+---------------------+-----------------------------------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|snapshot_id
|reference_snapshot_id|reference_snapshot_timestamp_millis|status|data_file
|
+-------------------+---------------------+-----------------------------------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|4341802652067419299|4341802652067419299 |1725425584897
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4341802652067419299|971355134943407989 |1725425585080
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|971355134943407989 |971355134943407989 |1725425585080
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|971355134943407989 |8947587060401590385 |1725425585198
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|4341802652067419299|8947587060401590385 |1725425585198
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|8947587060401590385|8947587060401590385 |1725425585198
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-2-c6c84ab2-06b9-4d49-8239-460e448c3c35-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|8947587060401590385|2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-2-c6c84ab2-06b9-4d49-8239-460e448c3c35-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|971355134943407989 |2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|2421236859611202602|2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-3-d3748dc0-4708-4a16-820d-81e2214765c1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|4341802652067419299|2421236859611202602 |1725425585316
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-2-c6c84ab2-06b9-4d49-8239-460e448c3c35-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-1-65b2a2db-fc1d-44ee-85d5-2e73a7826d33-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-0-9d6311fb-28da-48e0-b9fa-c8dfd32748b1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [61]}, {1 -> [01 00 00 00], 2 -> [61]}, NULL,
[4], NULL, 0}|
|4324292129886250863|4324292129886250863 |1725425585433
|2 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-3-d3748dc0-4708-4a16-820d-81e2214765c1-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
|2788755331636060629|2788755331636060629 |1725425585574
|1 |{0,
file:/var/folders/vh/xgbp56d10b72cjzlng9h8krc0000gn/T/junit12937496981447592132/data/00000-4-9ba23c37-daaf-4e93-a9bb-5fd06b954fb2-0-00001.parquet,
PARQUET, 0, 1, 614, {1 -> 42, 2 -> 43}, {1 -> 1, 2 -> 1}, {1 -> 0, 2 -> 0},
{}, {1 -> [01 00 00 00], 2 -> [62]}, {1 -> [01 00 00 00], 2 -> [62]}, NULL,
[4], NULL, 0}|
+-------------------+---------------------+-----------------------------------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
