0. In first phase we are not doing chunking/fragmentation. And even this will
be option for
client.(https://cwiki.apache.org/confluence/display/GEODE/Message+Structure+and+Definition#MessageStructureandDefinition-Protocolnegotiation)
1. Are you refereeing websocket/spdy? But I think we are talking almost same
thing, may be push isPartialMessage flag with chunk length(Anthony's example
below) ?
2. That's the part of the problem. Even if you need to serialize the "String",
you need to write length first and then need to write serialized utf bytes. We
can implement chunked input stream and can de-serialize the object as it is
coming (DataSerializable.fromData(ChunkedStream)).
From: Jacob Barrett <jbarr...@pivotal.io>
To: dev@geode.apache.org; Hitesh Khamesra <hitesh...@yahoo.com>
Cc: Anthony Baker <aba...@pivotal.io>
Sent: Friday, May 5, 2017 7:29 AM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
On Thu, May 4, 2017 at 2:52 PM Hitesh Khamesra <hitesh...@yahoo.com.invalid>
wrote:
Basically, thread/layer should not hold any resources while serializing the
object or chunk. We should be able to see this flow (ms1-chunk1, msg2-chunk1,
ms1-chunk2, msg3-chunk, msg2-chunk2, so on ...)
Correct, but putting that in the message layer is not appropriate. The simple
solution is that the multiple channels can be achieved with multiple sockets.
The later optimization is to add a channel multiplexer layer between the
message and socket layers.
If we put it in the message layer, not only does it for the message to tackle
something it shouldn't be concerned with, reassembling itself, but it also
forces all implementors to tackle this logic up front. By layering we can
release without, implementors aren't forced into understanding the logic, and
later we can release the layers and the client can negotiate.
On other pdx note: to de-serialize the pdx we need length of serialized bytes,
so that we can read field offset from serialized stream, and then can read
field value. Though, I can imagine with the help of pdxType, we can interpret
serialized stream.
Yes, so today PDX serialization would be no worse, the PDX serializer would
have to buffer, but other may not have to. The length of the buffered PDX could
be used as the first chunk length and complete in single chunk. Although, I
suspect that amortized overhead of splitting the chunks will be nil anyway.
The point is that the message encoding of values should NOT have any unbounded
length fields and require long or many buffers to complete serialization. By
chunking you can accomplish this by not needing to buffer the whole stream,
just small (say 1k), chunks at a time to get the chunk length.
Buffers == Latency
-Jake
