osscm commented on PR #18187: URL: https://github.com/apache/iceberg/pull/18187#issuecomment-6838272056
Keeping this PR open; status update. This is a reference implementation of the index spec (#16961). The spec has been revised substantially since this PR was based on it (most recently on Nov 9), so the on-disk format here follows an earlier draft of the spec. I plan to realign the metadata, tracking-file and region-file layers with the spec once it settles, rather than chasing each revision. Separately, I have continued the engine side on a Spark 4.0 branch (not part of this PR): https://github.com/osscm/iceberg/tree/scalar_sec_index_spark40 . It adds a resolver extracted from the scan builder, driver-side caching of index metadata, and handling for lookups that match nothing (an empty result prunes the scan instead of falling back to a full scan), plus a benchmark harness. cc @huaxingao @pvary -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
