Hi Anthony:
Help me to understand data chunking here?
>> bytes => arbitrary byte[] that can be chunked
Message => MessageHeader MessageBody
So lets say we want to send long byte[] into two chunks, then we will send two
messages? And other side will combine those two messages using "correlationId" ?
Thanks.
HItesh
________________________________
From: Anthony Baker <aba...@pivotal.io>
To: dev@geode.apache.org
Cc: Hitesh Khamesra <hitesh...@yahoo.com>
Sent: Wednesday, May 3, 2017 5:42 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
> On May 3, 2017, at 1:33 PM, Galen M O'Sullivan <gosulli...@pivotal.io> wrote:
>
> On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra <
> hitesh...@yahoo.com.invalid> wrote:
>
>> Absolutely its a implementation detail.
>>
> This doesn't answer Dan's comment. Do you think fragmentation should be
> taken care of by the TCP layer or the protocol should deal with it
> specifically?
There’s some really good feedback and discussion in this thread! Here are a
few thoughts:
1) Optional metadata should be used for fields that are generally applicable
across all messages. If a metadata field is required or only applies to a
small set of messages, it should become part of a message definition. Of
course there’s some grey area here.
2) I think we should pull out the message fragmentation support to avoid some
significant complexity. We can later add a fragmentation / envelope layer on
top without disrupting the current proposal. I do think we should add the
capability for chunking data (more on that below).
3) I did not find any discussion of message pipelining (queuing multiple
requests on a socket without waiting for a response) or out-of-order responses.
What is the plan for these capabilities and how will that affect consistency?
What about retries?
4) Following is an alternative definition with these characteristics:
- Serialized data can either be primitive or encoded values. Encoded values
are chunked as needed to break up large objects into a series of smaller parts.
- Because values can be chunked, the size field is removed. This allows the
message to be streamed to the socket incrementally.
- The apiVersion is removed because we can just define a new body type with a
new apiId (e.g. GetRequest2 with aipId = 1292).
- The GetRequest tells the server what kind of encoding the client is able to
understand.
- The metadata map is not used for fields that belong in the message body. I
think it’s much easier to write a spec without if statements :-)
Message => MessageHeader MessageBody
MessageHeader => correlationId metadata
correlationId => integer
metadata => count (key value)*
count => integer
key => string
value => string
MessageBody => apiId body
apiId => integer
body => (see specific definitions)
GetRequest => 0 acceptEncoding key
0 => the API id
acceptEncoding => (define some encodings for byte[], JSON, PDX, *, etc)
key => EncodedValue
GetResponse => 1 value
1 => the API id
value => EncodedValue
PutRequest => 2 eventId key value
2 => the API id
eventId => clientId threadId sequenceId
clientId => string
threadId => integer
sequenceId => integer
key => EncodedValue
value => EncodedValue
EncodedValue => encoding (boolean | integer | number | string | ((length
bytes)* 0))
encoding => (define some encodings for byte[], JSON, PDX, *, etc)
boolean => TRUE or FALSE
integer => a signed integer value
number => a decimal value corresponding to IEEE 754
string => UTF-8 text
bytes => arbitrary byte[] that can be chunked
Anthony
