ZENOTME opened a new pull request, #1014:
URL: https://github.com/apache/iceberg-rust/pull/1014
For read or write in iceberg-rust, we interoperate with arrow RecordBatch.
There are similar project tasks when reading and write to project RecordBatch:
- In equality delete, we need to project record batch according to field id
- in read, the user may scan part of the column and we also need to project
record batch according to field id
- for projects in reading, the iceberg also defines project rules to fulfill
the value when a column is missing. Like init-default
For now, these tasks are implemented in different structs,
`RecordBatchTransform`, and `RecordBatchProjector` and there are redundant
codes. After we introduce the `SchemaWithPartner` visitor in #731, we can
unify the implementation under the "visitor" framework.
The project task is generally separated into two parts:
1. accessor: retrieve the array of partners.
We can custom accessor to achieve init-default. E.g. when fail to
retrieve the array, accessor can construct the array according to init-default.
And we also can custom accessor to achieve name-map.
2. visitor: do some convert task.
We can customize visitors to do the type of promotion here.
The PR try this work to refactor our `RecordBatchTransform`, and
`RecordBatchProjector`. And it also complete
https://github.com/apache/iceberg-rust/issues/405#issue-2354723597 using
[ArrowProjectionVisitor](https://github.com/apache/iceberg-python/blob/afdfa351119090f09d38ef72857d6303e691f5ad/pyiceberg/io/pyarrow.py#L1135)
as @liurenjie1024 mention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
