unikdahal opened a new issue, #6719: URL: https://github.com/apache/datafusion-comet/issues/6719
### Describe the bug `CometScanRule.isIcebergMetadataTable` matches the table name with `endsWith` against the metadata table names (`files`, `snapshots`, `history`, `entries`, `manifests`, `partitions`, ...) with no separator. Any ordinary Iceberg table whose name ends with one of those words is treated as a metadata table and falls back to Spark with `Iceberg Metadata tables are not supported`, for example `user_files`, `order_history`, `daily_partitions`, or a table named `snapshots`. ```scala val name = scanExec.table.name().toLowerCase(Locale.ROOT) metadataTableSuffix.exists(name.endsWith) ``` ### Steps to reproduce ```sql CREATE TABLE cat.db.user_files (id INT, name STRING) USING iceberg; INSERT INTO cat.db.user_files VALUES (1, 'a'), (2, 'b'); SELECT * FROM cat.db.user_files; ``` With `spark.comet.scan.icebergNative.enabled=true` and `spark.comet.explain.fallback.enabled=true` the scan is reported as: ``` BatchScan cat.db.user_files [COMET: Iceberg Metadata tables are not supported] ``` The same table named `users` gets `CometIcebergNativeScan`. ### Expected behavior Only real metadata tables (`cat.db.user_files.files`, `...metadata.json#files`) should fall back. ### Additional context Proposed fix: recognize a metadata table by its Iceberg type. Every metadata table extends `org.apache.iceberg.BaseMetadataTable`, and the table is reachable through the existing `IcebergReflection.getTable(scan)`. Fall back to a whole-segment name match (`.files`, `#files`) only when the table cannot be reached. Happy to open a PR with a test covering `user_files`, `snapshots` and `order_history` plus a real `.files` read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
