zhuqi-lucas commented on issue #21290: URL: https://github.com/apache/datafusion/issues/21290#issuecomment-4169918791
Thanks for the response. After more investigation, I confirmed that vanilla DF 52's `ParquetOpener` + `PhysicalExprAdapterFactory` does handle the cases I described (Utf8→Date32 coercion, nullability adaptation). Our issues were caused by our custom `ExecutionPlanFactory` that creates `ParquetSource` directly — we had to add workarounds (force nullable table schema, custom `replace_schema` logic in our fork's opener) that introduced cascading side effects. The root of our pain was adapting our custom code to the new architecture, not a bug in vanilla DF 52. One remaining question: is there an equivalent of `SchemaAdapterFactory` as an extension point in DF 52? We used it in DF 51 to plug in custom schema adaptation. `PhysicalExprAdapterFactory` seems to be the replacement but it only adapts expressions, not batch data. Is there a recommended way for custom `TableProvider` implementations to do batch-level schema adaptation in DF 52? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
