sunchao commented on code in PR #5663:
URL: https://github.com/apache/datafusion-comet/pull/5663#discussion_r3954190192
##########
spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala:
##########
@@ -1290,6 +1293,146 @@ class CometIcebergWriteActionSuite
}
}
+ // A one-byte target file size makes the rolling writer finalize a file per
batch, and a two-row
+ // Comet batch size means the task has handed several batches to the writer
before the UDF
+ // throws on id 7. iceberg-java's writer abort deletes such files; the
native path must too.
+ test("native acceleration: a failed task deletes the data files it already
finalized") {
+ assumeNativeAcceleration()
+ withIcebergCatalog { warehouseDir =>
+ val session = spark
+ import session.implicits._
+ (1 to 10)
+ .map(i => (i, s"r$i", i.toDouble))
+ .toDF("id", "region", "amount")
+ .coalesce(1)
+ .createOrReplaceTempView("cleanup_src")
+ spark.udf.register(
+ "boom_on_seven_cleanup",
+ (id: Int) => {
+ if (id == 7) throw new RuntimeException("boom")
+ id
+ })
+ val rollingProps = Some("'write.target-file-size-bytes'='1'")
+
+ withNativeEnabled(withSQLConf(CometConf.COMET_BATCH_SIZE.key -> "2") {
+ // Control: the same source and settings without the failure roll into
several files, so
+ // the failing run below really does have finalized files to clean up.
+ createTable(
+ warehouseDir,
+ "cleanup_control",
+ partitionSpec = "",
+ properties = rollingProps)
+ val controlPlans = capturePlans {
Review Comment:
### Correctness
#### [P1] Adapt the new cleanup tests to the shared plan helper
The merged base now defines `capturePlans(spark: SparkSession)(action: =>
Unit)` in `CometIcebergTestBase`, so this old call shape supplies the INSERT
DataFrame where the SparkSession is required. The new `captureFailedPlans`
helper at lines 1962–1964 also uses `QueryExecutionListener` and
`QueryExecution` after their imports were removed. Please pass `spark` here and
restore those imports. Current [Spark 3.5
scans](https://github.com/apache/datafusion-comet/actions/runs/34132536375/job/101782225517)
and [Spark 4.1
scans](https://github.com/apache/datafusion-comet/actions/runs/34132536375/job/101782225562)
both fail test compilation with these same four errors, so the cleanup tests
cannot run. I verified the CI merge has the exact assigned base/head parents
and the same source tree as this head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
