nssalian commented on issue #18430: URL: https://github.com/apache/iceberg/issues/18430#issuecomment-6079273065
Thanks for finding this. This behavior is expected in the sense that shredded Variant is not yet supported by the vectorized reader, so projected shredded Variant columns should continue to fall back to the row reader. The bug here is the unprojected case. If the Variant column is not part of the expected projection, it should be skipped so the rest of the scan can remain vectorized. Right now the reader builder still walks the shredded Variant subtree and hits the unsupported vectorized path. Your fix looks like the right short-term fix to me: skip creating the Variant reader when the field is not projected, while preserving the existing fallback for projected shredded Variant columns. We can get that in for now, I'm not able to help immediately but someone should help review and merge it for now. Proper vectorized reconstruction for shredded Variant is still follow-up work. I'm hoping to get that in a few weeks. Awaiting the struct PR to get more eyes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
