[
https://issues.apache.org/jira/browse/KAFKA-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Nee updated KAFKA-17825:
-------------------------------
Description:
We've noticed that using the ByteBufferDeserializer can yield a different byte
array length compare to the deserializer from 3.5.2. This is attributed by
KIP-863, in particular, the old deserializer truncated the byte array starting
from
`buffer.position() + buffer.arrayOffset() + offset` using `Utils.toArray`
Whereas the current implementation is a passthrough.
This can be reproduced using the
{code:java}
KafkaConsumerProducerDemo.java{code}
by changing the type to <Integer, ByteBuffer> and perform a print after poll.
For example, the producer produces a record [0, test0] (key is an int, "test0"
is a 5 bytes long string, converted to byte buffer using
{code:java}
ByteBuffer.wrap(value.getBytes()){code}
Prior to KIP-863 we see the following after polling the record from consumer:
3.5.2: test0
3.6.0:
{code:java}
?$���y�NNޅ�-p�=�����NAc�����8D���8D���������������
test0{code}
And if you analyze the ByteBuffer post 3.6.0, we can see the current offset is
at 140 with array length of 149.
[~LSK] - since you wrote the kip and did the implementation, can you address
this ?
was:
We've noticed that using the ByteBufferDeserializer can yield a different byte
array length compare to the deserializer from 3.5.2. This is attributed by
KIP-863, in particular, the old deserializer truncated the byte array starting
from
`buffer.position() + buffer.arrayOffset() + offset` using `Utils.toArray`
Whereas the current implementation is a passthrough.
This can be reproduced using the
{code:java}
KafkaConsumerProducerDemo.java{code}
by changing the type to <Integer, ByteBuffer> and perform a print after poll.
For example, the producer produces a record [0, test0] (key is an int, "test0"
is a 5 bytes long string, converted to byte buffer using
{code:java}
ByteBuffer.wrap(value.getBytes()){code}
Prior to KIP-863 we see the following after polling the record from consumer:
3.5.2: test0
3.6.0:
{code:java}
?$���y�NNޅ�-p�=�����NAc�����8D���8D���������������
test0{code}
And if you analyze the ByteBuffer post 3.6.0, we can see the current position
is at 140 with array length of 149.
[~LSK] - since you wrote the kip and did the implementation, can you address
this ?
> ByteBufferDeserializaer's array size can be inconsistent with the older
> version
> -------------------------------------------------------------------------------
>
> Key: KAFKA-17825
> URL: https://issues.apache.org/jira/browse/KAFKA-17825
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 3.6.0, 3.7.0, 3.6.1, 3.6.2, 3.8.0, 3.7.1, 3.9.0, 3.8.1,
> 3.9.1
> Reporter: Philip Nee
> Assignee: LinShunkang
> Priority: Blocker
>
> We've noticed that using the ByteBufferDeserializer can yield a different
> byte array length compare to the deserializer from 3.5.2. This is attributed
> by KIP-863, in particular, the old deserializer truncated the byte array
> starting from
> `buffer.position() + buffer.arrayOffset() + offset` using `Utils.toArray`
>
> Whereas the current implementation is a passthrough.
>
> This can be reproduced using the
> {code:java}
> KafkaConsumerProducerDemo.java{code}
> by changing the type to <Integer, ByteBuffer> and perform a print after poll.
> For example, the producer produces a record [0, test0] (key is an int,
> "test0" is a 5 bytes long string, converted to byte buffer using
> {code:java}
> ByteBuffer.wrap(value.getBytes()){code}
> Prior to KIP-863 we see the following after polling the record from consumer:
> 3.5.2: test0
> 3.6.0:
> {code:java}
> ?$���y�NNޅ�-p�=�����NAc�����8D���8D���������������
> test0{code}
>
> And if you analyze the ByteBuffer post 3.6.0, we can see the current offset
> is at 140 with array length of 149.
>
> [~LSK] - since you wrote the kip and did the implementation, can you address
> this ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
