mbutrovich commented on code in PR #3842:
URL: https://github.com/apache/datafusion-comet/pull/3842#discussion_r3034248216
##########
spark/src/test/scala/org/apache/spark/sql/comet/CometTaskMetricsSuite.scala:
##########
@@ -91,4 +99,78 @@ class CometTaskMetricsSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
}
}
}
+
+ test("native_datafusion scan reports task-level input metrics matching
Spark") {
+ val totalRows = 10000
+ withTempPath { dir =>
+ val rng = new scala.util.Random(42)
+ spark
+ .createDataFrame((0 until totalRows).map(_ => (rng.nextInt(),
rng.nextLong())))
Review Comment:
Can we use a smaller range for the random values, or just a shuffle from 0
to `totalRows`? That way we'd know exactly the amount of data we should get
back. Right now it's likely selecting every row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
