dwsmith1983 opened a new issue, #5541:
URL: https://github.com/apache/datafusion-comet/issues/5541
### Describe the bug
CometScanRule.validateIcebergFileScanTasks checks Iceberg file schemes
against a hardcoded set (file, s3, s3a, gs, gcs, oss, abfss, abfs, wasbs, wasb
at CometScanRule.scala:1082), and its fallback message repeats that list to
users. But the Iceberg execution path resolves storage through its own factory
(iceberg_scan.rs storage_factory_for), which supports only file, s3, s3a, gs,
oss via OpenDAL. And the JNI probe every other scheme gate delegates to
(isNativelyReadableScheme → object_store::ObjectStoreScheme::parse) recognizes
a third, different set.
Net effect: an Iceberg table under gcs, abfs, abfss, wasb, or wasbs passes
validation, gets claimed, and every task dies with CometNativeException:
Unsupported storage scheme: ... — while stock Spark reads the same table fine.
Meanwhile oss actually works at execution despite the JNI probe saying it's
unsupported, so the three lists disagree in both directions.
### Steps to reproduce
No cloud account needed — back the scheme with local disk the way
ParquetReadFromFakeHadoopFsSuite does:
// register a local-backed filesystem for the wasb scheme
spark.conf.set("spark.hadoop.fs.wasb.impl",
"org.apache.hadoop.fs.RawLocalFileSystem")
// hadoop catalog with a wasb warehouse
// spark.sql.catalog.hadoop_catalog = org.apache.iceberg.spark.SparkCatalog
// spark.sql.catalog.hadoop_catalog.type = hadoop
// spark.sql.catalog.hadoop_catalog.warehouse =
wasb://fake-container/tmp/warehouse
sql("CREATE TABLE hadoop_catalog.t (id INT, s STRING) USING iceberg")
sql("INSERT INTO hadoop_catalog.t VALUES (1,'a'),(2,'b'),(3,'c')")
sql("SELECT * FROM hadoop_catalog.t").collect()
With spark.comet.enabled=false: [1,a],[2,b],[3,c].
With Comet + spark.comet.scan.icebergNative.enabled=true (plan shows
CometIcebergNativeScan ... wasb://fake-container/...): every task fails with
org.apache.comet.CometNativeException: Unsupported storage scheme: wasb
(Ensure spark.comet.libhdfs.schemes doesn't include wasb, since that conf
routes schemes elsewhere.)
### Expected behavior
Schemes the Iceberg native path can't execute should be declined at plan
time so the scan falls back to Spark, and the fallback message should list
what's actually supported. One source of truth: the validator should ask the
same layer that executes — either expose the Iceberg storage factory's
supported set through the existing JNI probe pattern, or derive the check from
it directly — rather than maintaining a third hand-written list.
### Additional context
The claimed-but-fails set on this path is {gcs, abfs, abfss, wasb, wasbs};
oss works at execution but is rejected by the generic JNI probe, so aligning
the lists fixes both directions. Same drift class as the scheme/config gates
reworked in #5365.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
