gerashegalov commented on code in PR #18709:
URL: https://github.com/apache/iceberg/pull/18709#discussion_r4513865500
##########
spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadMetrics.java:
##########
@@ -46,6 +52,80 @@ public void removeTables() {
sql("DROP TABLE IF EXISTS %s", tableName);
}
+ @TestTemplate
+ public void testEffectiveSplitTargetSize() throws NoSuchTableException {
+ sql(
+ "CREATE TABLE %s (id BIGINT) USING iceberg TBLPROPERTIES
('read.split.target-size'='134217728')",
+ tableName);
+ spark.range(100).coalesce(1).writeTo(tableName).append();
+
+ withSQLConf(
+ ImmutableMap.of(
+ SparkSQLProperties.READ_ADAPTIVE_SPLIT_SIZE_ENABLED,
+ "true",
+ SparkSQLProperties.READ_ADAPTIVE_SPLIT_SIZE_PARALLELISM,
+ "1000"),
+ () -> assertEffectiveSplitTargetSize(16L * 1024 * 1024));
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.READ_ADAPTIVE_SPLIT_SIZE_ENABLED,
"false"),
+ () -> assertEffectiveSplitTargetSize(128L * 1024 * 1024));
+
+ withSQLConf(
+ ImmutableMap.of(SparkSQLProperties.READ_ADAPTIVE_SPLIT_SIZE_ENABLED,
"true"),
+ () -> {
+ Dataset<Row> df =
+ spark
+ .read()
+ .format("iceberg")
+ .option(SparkReadOptions.SPLIT_SIZE, String.valueOf(32L *
1024 * 1024))
+ .load(tableName);
+ assertEffectiveSplitTargetSize(df, 32L * 1024 * 1024);
+ });
+ }
+
+ @TestTemplate
+ public void testCustomMetricsAreInstantiableBySpark() {
Review Comment:
On reflection I moved this back into its own test,
`customMetricsAreLoadableBySpark` (1fb30381b). AGENTS.md asks for "One behavior
per test method. No omnibus tests asserting several unrelated things" and
"Write the most direct test for the bug". Spark being able to load the metric
classes is a separate behavior from the V1/V2 metric values. Keeping it
separate means a regression like the original private nested class fails one
clearly named test instead of both read-metrics tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
