bharos commented on code in PR #12327:
URL: https://github.com/apache/iceberg/pull/12327#discussion_r1962463880
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java:
##########
@@ -858,6 +858,23 @@ public static String quotedFullIdentifier(String
catalogName, Identifier identif
.quoted();
}
+ public static org.apache.spark.sql.execution.datasources.PartitionSpec
getInferredSpec(
+ SparkSession spark, Path rootPath) {
+ FileStatusCache fileStatusCache = FileStatusCache.getOrCreate(spark);
+ InMemoryFileIndex fileIndex =
+ new InMemoryFileIndex(
Review Comment:
Add files will do this InMemoryFileIndex calculation twice (first it will do
here to infer the spec with option.empty()) and later it will be called in
getPartitions (see L902) to list the files. I assume since fileStatusCache is
passed with same spark session it will not do recomputation?
Else I will calculate the InMemoryFileIndex in AddFilesProcedure and infer
schema there, and pass it along to getPartitions()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
