Fokko commented on issue #401: URL: https://github.com/apache/iceberg-python/issues/401#issuecomment-1944638806
Hey @srilman sorry for not replying earlier. It slipped somewhere through the cracks, thanks for pinging me! > Is there a recommended way to getting the base / original schema or schema-id of a data file in a FileScanTask returned during FileTableScan.plan_files? This is useful to determine what kind of schema evolution occurred with the subset of files we are reading, and group files together with the same schemas for reads. Unfortunately, no :( > From there, we can get the associated schema per snapshot. A snapshot can have files with multiple schemas. See the example in https://github.com/duckdb/duckdb_iceberg/issues/40 > Is this a logical approach, or is there a better way to get the original schema? Happy to open a PR to integrate this into FileTableScan if it would be useful! The only source of truth is in the Parquet file. The crux is that you don't want to do name-based projection, but field-id-based projection. Names can change over time, but field-id won't. So if you filter on a specific field-id, that will always work correctly. The same goes when reading parquet files, it can be that the names change over time, and you don't want to rewrite petabytes of data, so you can leave them as is. When reading, you want to make the field-ids to the current schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
