dramaticlly commented on code in PR #18406:
URL: https://github.com/apache/iceberg/pull/18406#discussion_r4212955454
##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java:
##########
@@ -2028,6 +2029,28 @@ public void testSnapshotProperty() {
assertThat(table.currentSnapshot().summary()).containsKeys(commitMetricsKeys);
}
+ @TestTemplate
Review Comment:
can we add a test to cover if override session or explicit snapshot property
collide with reserved `added-data-files`, just to pin down the behavior.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSnapshotUpdateSparkAction.java:
##########
@@ -21,14 +21,21 @@
import java.util.Map;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.PropertyUtil;
import org.apache.spark.sql.SparkSession;
+import scala.collection.JavaConverters;
abstract class BaseSnapshotUpdateSparkAction<ThisT> extends
BaseSparkAction<ThisT> {
private final Map<String, String> summary = Maps.newHashMap();
protected BaseSnapshotUpdateSparkAction(SparkSession spark) {
super(spark);
+ summary.putAll(
+ PropertyUtil.propertiesWithPrefix(
+ JavaConverters.mapAsJavaMap(spark.conf().getAll()),
+ SparkSQLProperties.SNAPSHOT_PROPERTY_PREFIX));
Review Comment:
on a tangentially related note, I think we might drop the `snapshotProperty`
on RevemoDanglingDeletesSparkAction before but not anymore after this change.
```
SparkActions.get(spark).rewriteDataFiles(table)
.option(RewriteDataFiles.REMOVE_DANGLING_DELETES, "true")
.snapshotProperty("audit-id", "A")
.execute();
```
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSnapshotUpdateSparkAction.java:
##########
@@ -21,14 +21,21 @@
import java.util.Map;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.spark.SparkSQLProperties;
+import org.apache.iceberg.util.PropertyUtil;
import org.apache.spark.sql.SparkSession;
+import scala.collection.JavaConverters;
abstract class BaseSnapshotUpdateSparkAction<ThisT> extends
BaseSparkAction<ThisT> {
private final Map<String, String> summary = Maps.newHashMap();
protected BaseSnapshotUpdateSparkAction(SparkSession spark) {
super(spark);
+ summary.putAll(
+ PropertyUtil.propertiesWithPrefix(
+ JavaConverters.mapAsJavaMap(spark.conf().getAll()),
+ SparkSQLProperties.SNAPSHOT_PROPERTY_PREFIX));
Review Comment:
also I think this duplicates with SparkWriteConf to retrieve the session
level snapshot properties, might consider a common method in SparkUtil?
```java
public static Map<String, String> sessionSnapshotProperties(SparkSession
spark) {
return PropertyUtil.propertiesWithPrefix(
JavaConverters.mapAsJavaMap(spark.conf().getAll()),
SparkSQLProperties.SNAPSHOT_PROPERTY_PREFIX);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
