raghav-reglobe commented on PR #63192: URL: https://github.com/apache/doris/pull/63192#issuecomment-5055274704
We've been running this branch rebased onto current master (ab043a43) against Iceberg V3 tables with shredded VARIANT parquet, and hit an interaction with the new file scanner: with `enable_file_scanner_v2=true` (the master default), any scan touching a VARIANT-bearing file fails at open with ``` [CORRUPTION]Failed to open parquet file: Metadata contains Thrift LogicalType that is not recognized ``` format_v2's footer path goes through arrow parquet-cpp (17.0.0 in thirdparty), which predates the parquet VARIANT LogicalType — and this PR's read support lives in the v1 reader tree, so v2 has no path for these files even past the footer. With `enable_file_scanner_v2=false` everything in this branch works well for us (bracket access, predicates on shredded columns, extraction — we've verified against fairly large shredded datasets). Is scanner-v2 integration planned as part of this PR? Happy to help — either the small interim guard (routing VARIANT-bearing scans to v1 in `_should_use_file_scanner_v2`, same shape as the existing WAL/JNI/Lance exclusions, so the default-on v2 doesn't crash) or contributing toward proper v2 reader support if you'd take it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
