RussellSpitzer commented on code in PR #12319:
URL: https://github.com/apache/iceberg/pull/12319#discussion_r1960708406
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:
##########
@@ -1085,4 +1088,43 @@ private ExecutorService getService() {
return service;
}
}
+
+ /**
+ * Returns the first partition spec in an IcebergTable that shares the same
names and ordering as
+ * the partition columns in a given Spark Table. Throws an error if not found
+ */
+ private static PartitionSpec findCompatibleSpec(
+ Table icebergTable, SparkSession spark, String sparkTable) throws
AnalysisException {
+ List<String> parts =
Lists.newArrayList(Splitter.on('.').limit(2).split(sparkTable));
+ String db = parts.size() == 1 ? "default" : parts.get(0);
+ String table = parts.get(parts.size() == 1 ? 0 : 1);
+
+ List<String> sparkPartNames =
+ spark.catalog().listColumns(db, table).collectAsList().stream()
+ .filter(org.apache.spark.sql.catalog.Column::isPartition)
+ .map(org.apache.spark.sql.catalog.Column::name)
+ .map(name -> name.toLowerCase(Locale.ROOT))
+ .collect(Collectors.toList());
+
+ for (PartitionSpec icebergSpec : icebergTable.specs().values()) {
+ boolean allIdentity =
+ icebergSpec.fields().stream().allMatch(field ->
field.transform().isIdentity());
Review Comment:
Yeah I was trying to be lazy here, I guess just ignoring void transforms
isn't too complicated. I was trying to limit the scope of what I'm fixing but
that makes sense to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
