[
https://issues.apache.org/jira/browse/KAFKA-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852155#comment-17852155
]
Greg Harris commented on KAFKA-16257:
-------------------------------------
Hi [~fanyan] Here's my concerns:
1. The SchemaProjector is static, but the plugin.path configuration (and
Plugins class, PluginClassLoaders, etc) are instance-oriented, and multiple
Plugins instances can exist at the same time in a JVM.
2. We're currently about to implement multi-version support:
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-891%3A+Running+multiple+versions+of+Connector+plugins]
which will greatly complicate how to resolve what version of the plugins to
use. For example, if an SMT defines a new logical type, and two versions of
that SMT are present in the pipeline, which projector should be used?
3. One of the persistent classloading flaws that we encounter is an incorrect
value being returned from Thread#getContextClassLoader(). For example, when we
call AvroConverter methods, we ensure that the converter's PluginClassLoader is
set as the context classloader. If the FileSinkConnector calls
SchemaProjector#project, it will usually have the connector's PluginClassLoader
as the context classloader. That thread would need to switch to the
AvroConverter's class loader before safely calling AvroConverter code.
4. There are other issues with calling pluggable code, such as handling
exceptions, and indefinite blocking (e.g. infinite loops, or deadlocks).
Point 1 is I think the biggest challenge that we'll need to overcome for this
change specifically.
Point 2 is a complication that makes plugins-calling-other-plugins more
challenging than it already is, and that we should design around.
Points 3 and 4 I'm interested in improving via KAFKA-14670, and just need some
mindfulness during implementation here.
> SchemaProjector should be extensible to logical types
> -----------------------------------------------------
>
> Key: KAFKA-16257
> URL: https://issues.apache.org/jira/browse/KAFKA-16257
> Project: Kafka
> Issue Type: New Feature
> Components: connect
> Reporter: Greg Harris
> Assignee: Fan Yang
> Priority: Minor
> Labels: needs-kip
>
> The SchemaProjector currently only supports projecting primitive Number
> types, and cannot handle common logical types as have proliferated in the
> Connect ecosystem.
> The SchemaProjector or a replacement should have the ability to extend it's
> functionality to support these logical types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
