RussellSpitzer commented on code in PR #12327:
URL: https://github.com/apache/iceberg/pull/12327#discussion_r1968556347
##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java:
##########
@@ -635,6 +635,67 @@ public void
addFilteredPartitionsToPartitionedWithNullValueFilteringOnDept() {
sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void addFileTableOldSpecDataAfterPartitionSpecEvolved() {
+ createPartitionedFileTable("parquet");
+ createIcebergTable(
+ "id Integer, name String, dept String, subdept String",
+ "PARTITIONED BY (id, dept, subdept)");
+ sql("ALTER TABLE %s DROP PARTITION FIELD dept", tableName);
+ sql(
+ "ALTER TABLE %s DROP PARTITION FIELD subdept",
+ tableName); // This spec is matching with the input data which is
partitioned just by "id"
+ sql("ALTER TABLE %s ADD PARTITION FIELD subdept", tableName);
+
+ if (formatVersion == 1) {
+ assertThatThrownBy(
+ () ->
+ scalarSql(
+ "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
+ catalogName, tableName, fileTableDir.getAbsolutePath()))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining(
+ String.format(
+ "Cannot add data files to target table %s because that table
is partitioned and contains non-identity partition transforms which will not be
compatible.",
+ tableName));
+ return;
+ }
+
+ List<Object[]> result =
+ sql(
+ "CALL %s.system.add_files(table => '%s', source_table =>
'`parquet`.`%s`')",
+ catalogName, tableName, fileTableDir.getAbsolutePath());
+
+ assertOutput(result, 8L, 4L);
+ assertEquals(
+ "Iceberg table contains correct data",
+ sql("SELECT id, name, dept, subdept FROM %s ORDER BY id",
sourceTableName),
+ sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
Review Comment:
Please also add in a check where we check the metadata tables and show that
the correct spec id has been set for the newly added files. You can get the
specId programmatically after you make it above with table.specId() and then
assert that it is set for the new files here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
