gustavoatt commented on code in PR #7120:
URL: https://github.com/apache/iceberg/pull/7120#discussion_r1149502297
##########
spark/v3.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestPartitionedWrites.java:
##########
@@ -157,4 +158,64 @@ public void testViewsReturnRecentResults() {
ImmutableList.of(row(1L, "a"), row(1L, "a")),
sql("SELECT * FROM tmp"));
}
+
+ @Test
+ public void testWriteWithOutputSpec() throws NoSuchTableException {
+ Table table = validationCatalog.loadTable(tableIdent);
+
+ final int originalSpecId = table.spec().specId();
+ table.updateSpec().addField("data").commit();
+
+ table.refresh();
+ sql("REFRESH TABLE %s", tableName);
+
+ // By default, we write to the current spec.
+ sql("INSERT INTO TABLE %s VALUES (10, 'a')", tableName);
+
+ List<Object[]> expected = ImmutableList.of(row(10L, "a",
table.spec().specId()));
+ assertEquals(
+ "Rows must match",
+ expected,
+ sql("SELECT id, data, _spec_id FROM %s WHERE id >= 10 ORDER BY id",
tableName));
+
+ // Output spec ID should be respected when present.
+ List<SimpleRecord> data =
+ ImmutableList.of(new SimpleRecord(11, "b"), new SimpleRecord(12, "c"));
+ spark
+ .createDataFrame(data, SimpleRecord.class)
+ .toDF()
+ .writeTo(tableName)
+ .option("output-spec-id", Integer.toString(originalSpecId))
+ .append();
+
+ expected =
+ ImmutableList.of(
+ row(10L, "a", table.spec().specId()),
+ row(11L, "b", originalSpecId),
+ row(12L, "c", originalSpecId));
+ assertEquals(
+ "Rows must match",
+ expected,
+ sql("SELECT id, data, _spec_id FROM %s WHERE id >= 10 ORDER BY id",
tableName));
Review Comment:
Done. Added it.
I had to skip that check on Spark 3.3. for Wap branch. There is a bug
reading the `.partitions` table on a WAP branch so I just skipped that check
(we still check that each row has the correct spec_id).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
