dwsmith1983 commented on PR #5654: URL: https://github.com/apache/datafusion-comet/pull/5654#issuecomment-5731467619
> Could you and @ErikBPF work out which cases each of you is claiming, and in what order they land? Order as ErikBPF said above: this PR first, #5786 adapts after. On the split: #6004 now declines at planning any requested schema that repeats an id, so I have dropped the equal-schema rationale from the tests here and they request a single id that the file carries twice. What this PR still adds is native raising Spark's error when a requested id is ambiguous in the file, last-wins exact-name lookup like Spark's `.toMap`, the id shield against a stray same-name column, one list-type definition across the mapping paths, and typed Spark errors surviving the JNI cause chain. #5786 claims byte-identical duplicate names that the decoder cannot represent. > I would expect a small check, that checks all the plan nodes and related schema and if the struct found with duplicated name, then fallback That check is what #6004 landed for field ids, and it stays. The reader-side error covers the case a plan-level check cannot see, an id the requested schema names once that the file carries twice. If you would rather keep one mechanism, I can trim the footer check to that file-side case; the adapter path has to stay for the Spark error text. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
