SagarJadhav30 commented on issue #57882:
URL: https://github.com/apache/doris/issues/57882#issuecomment-3707780110
This looks like a Spark-Doris-Connector issue where pushed-down filter or
projection metadata becomes inconsistent during write, resulting in an invalid
column reference (null) in the generated Doris query plan
error is query plan failed
Unknown column 'null' in 'table list'
and .option("doris.filter", "cust_no = '111'")
This appears to be a Spark-Doris-Connector issue rather than a Spark SQL
problem. The error originates from Doris query planning, and only occurs when
the source is Doris and doris.filter is used. When reading via JDBC, the same
write succeeds, which suggests the dataset schema itself is valid. My guess is
that filter or projection metadata from the Doris scan is leaking into the
write path, resulting in an unresolved column being serialized as null in the
generated Doris query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
