Guosmilesmile opened a new issue, #18142:
URL: https://github.com/apache/iceberg/issues/18142
### Feature Request / Improvement
Add Flink Lookup Join support to Iceberg with an incremental approach:
1. **Memory-only full cache**
- Use a heap-based cache with projection and filter pushdown.
- Track the snapshot used to build the cache.
- Load the cache once initially, targeting dimension tables that fit
comfortably in TaskManager heap.
2. **Periodic full reload**
- Refresh the cache at a configurable interval.
- Atomically swap in the new cache after a successful reload.
- Define an explicit failure policy that fails rather than silently
serving stale data.
3. **Incremental reload via snapshot deltas**
- Use Iceberg's incremental scan between the cached snapshot and the
current snapshot.
- Apply only the changes to the existing cache, so reload cost scales
with the change set rather than the entire table.
Large lookup tables can be addressed separately once the second index is
available. Potential approaches include RocksDB, FLIP-462 slicing, and async
lookup.
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
