mbutrovich commented on issue #4859: URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-5203957738
I just ran the tests from #23391 (that passed with that POC fix) against the latest DataFusion with https://github.com/apache/datafusion/pull/24090 in it and it did not resolve the issue: ``` 26/08/06 07:15:36 WARN shuffle/src/schema_align.rs: ShuffleWriter input schema mismatch on col[0] 'events': child produced List(Field { data_type: Struct([Field { name: "id", data_type: Int64, nullable: true }, Field { name: "payload", data_type: Utf8 }]), nullable: true }), catalyst declared List(Field { data_type: Struct([Field { name: "id", data_type: Int64 }, Field { name: "payload", data_type: Utf8 }]) }). Inserting a cast; please file the upstream function bug at https://github.com/apache/datafusion-comet/issues/4515. 802983 was not less than 401491 native scan read the whole nested column despite pruned ReadSchema (bytes_scanned: events.id=802983, events.id+payload=802983) ScalaTestFailureLocation: org.apache.comet.exec.CometNativeReaderSuite at (CometNativeReaderSuite.scala:1035) org.scalatest.exceptions.TestFailedException: 802983 was not less than 401491 native scan read the whole nested column despite pruned ReadSchema (bytes_scanned: events.id=802983, events.id+payload=802983) at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) 26/08/06 07:15:37 WARN shuffle/src/schema_align.rs: ShuffleWriter input schema mismatch on col[0] 'events': child produced List(Field { data_type: Struct([Field { name: "id", data_type: Int64, nullable: true }, Field { name: "inner", data_type: Struct([Field { name: "a", data_type: Int64, nullable: true }, Field { name: "blob", data_type: Utf8 }]) }]), nullable: true }), catalyst declared List(Field { data_type: Struct([Field { name: "id", data_type: Int64 }, Field { name: "inner", data_type: Struct([Field { name: "a", data_type: Int64 }, Field { name: "blob", data_type: Utf8 }]) }]) }). Inserting a cast; please file the upstream function bug at https://github.com/apache/datafusion-comet/issues/4515. 883241 was not less than 441620 native scan read unrequested nested leaves through array<struct<struct>> (bytes_scanned: events.inner.a=883241, events.inner.a+blob=883241) ScalaTestFailureLocation: org.apache.comet.exec.CometNativeReaderSuite at (CometNativeReaderSuite.scala:1075) org.scalatest.exceptions.TestFailedException: 883241 was not less than 441620 native scan read unrequested nested leaves through array<struct<struct>> (bytes_scanned: events.inner.a=883241, events.inner.a+blob=883241) at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471) at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231) ``` Looking into it today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
