szehon-ho commented on code in PR #8111:
URL: https://github.com/apache/iceberg/pull/8111#discussion_r1271758197
##########
spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewritePositionDeleteFiles.java:
##########
@@ -179,7 +179,7 @@ public void testIntPartition() throws Exception {
@Test
public void testDaysPartitionTransform() throws Exception {
- createTable("timestamp", "days(partition_col)");
+ createTable("timestamp", "days(`partition_col.type`)");
Review Comment:
Yes , can you please just add a new test? Its confusing if all the tests
now use `` and dot, it's unclear what they are testing for, its better if each
unit test is more targeted towards one fix/feature.
##########
spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewritePositionDeleteFiles.java:
##########
@@ -179,7 +179,7 @@ public void testIntPartition() throws Exception {
@Test
public void testDaysPartitionTransform() throws Exception {
- createTable("timestamp", "days(partition_col)");
+ createTable("timestamp", "days(`partition_col.type`)");
Review Comment:
I am aware, we need to change all the methods to take in additional
partitionCol parameter, with default value 'partitionCol'
For example:
```
private void createTable(String partitionType, String partitionCol, String
partitionTransform) {
sql(
"CREATE TABLE %s (id long, %s %s, c1 string, c2 string) "
+ "USING iceberg "
+ "PARTITIONED BY (%s) "
+ "TBLPROPERTIES('format-version'='2')",
tableName, partitionCol, partitionType, partitionTransform);
}
private void insertData(Function<Integer, ?> partitionValue, int
numDataFiles, String partitionCol) throws Exception {
for (int i = 0; i < numDataFiles; i++) {
Dataset<Row> df =
spark
.range(0, ROWS_PER_DATA_FILE)
.withColumn(partitionCol, lit(partitionValue.apply(i)))
.withColumn("c1", expr("CAST(id AS STRING)"))
.withColumn("c2", expr("CAST(id AS STRING)"));
appendAsFile(df);
}
}
private List<Object[]> records(String table, String partitionCol) {
return rowsToJava(
spark.read().format("iceberg").load(table).sort(partitionCol,
"id").collectAsList());
}
private void testDanglingDelete(String partitionCol) throws Exception {
... records(table, partitionCol);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
