And len 0 would indicate end of the message?
a. Now these two chunks will go continuous. b. If its PDX encoded then pdx header(1byte:pdxid 4byte:len 4byte:typeId) requires size of all pdx serialized bytes. So we know "size" of data upfront here. c. Lets say region value is just long byte[]: Then we have "size" to send the message. So in both cases we know the "size" of serialized bytes(payload). So possibly we don't need to chunk that message and let tcp take care of it? It seems we should walk through some more usecases to understand this better. Thanks. Hitesh ________________________________ From: Anthony Baker <aba...@pivotal.io> To: Hitesh Khamesra <hitesh...@yahoo.com> Cc: "dev@geode.apache.org" <dev@geode.apache.org> Sent: Thursday, May 4, 2017 11:20 AM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal There would be one Message containing a single MessageHeader and a single MessageBody. A PDX EncodedValue containing 1242 bytes that are chunked would look something like this: PDX 1000 byte[1000] 242 byte[242] 0 Anthony > On May 4, 2017, at 10:38 AM, Hitesh Khamesra <hitesh...@yahoo.com> wrote: > > Hi Anthony: > > Help me to understand data chunking here? > >>> bytes => arbitrary byte[] that can be chunked > > Message => MessageHeader MessageBody > > So lets say we want to send long byte[] into two chunks, then we will send > two messages? And other side will combine those two messages using > "correlationId" ? > > Thanks. > HItesh > > > > > ________________________________ > From: Anthony Baker <aba...@pivotal.io> > To: dev@geode.apache.org > Cc: Hitesh Khamesra <hitesh...@yahoo.com> > Sent: Wednesday, May 3, 2017 5:42 PM > Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal > > > > >> On May 3, 2017, at 1:33 PM, Galen M O'Sullivan <gosulli...@pivotal.io> wrote: >> >> On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra < >> hitesh...@yahoo.com.invalid> wrote: >> >>> Absolutely its a implementation detail. >>> >> This doesn't answer Dan's comment. Do you think fragmentation should be >> taken care of by the TCP layer or the protocol should deal with it >> specifically? > > There’s some really good feedback and discussion in this thread! Here are a > few thoughts: > > 1) Optional metadata should be used for fields that are generally applicable > across all messages. If a metadata field is required or only applies to a > small set of messages, it should become part of a message definition. Of > course there’s some grey area here. > > 2) I think we should pull out the message fragmentation support to avoid some > significant complexity. We can later add a fragmentation / envelope layer on > top without disrupting the current proposal. I do think we should add the > capability for chunking data (more on that below). > > 3) I did not find any discussion of message pipelining (queuing multiple > requests on a socket without waiting for a response) or out-of-order > responses. What is the plan for these capabilities and how will that affect > consistency? What about retries? > > 4) Following is an alternative definition with these characteristics: > > - Serialized data can either be primitive or encoded values. Encoded values > are chunked as needed to break up large objects into a series of smaller > parts. > - Because values can be chunked, the size field is removed. This allows the > message to be streamed to the socket incrementally. > - The apiVersion is removed because we can just define a new body type with a > new apiId (e.g. GetRequest2 with aipId = 1292). > - The GetRequest tells the server what kind of encoding the client is able to > understand. > - The metadata map is not used for fields that belong in the message body. I > think it’s much easier to write a spec without if statements :-) > > Message => MessageHeader MessageBody > > MessageHeader => correlationId metadata > correlationId => integer > metadata => count (key value)* > count => integer > key => string > value => string > > MessageBody => apiId body > apiId => integer > body => (see specific definitions) > > GetRequest => 0 acceptEncoding key > 0 => the API id > acceptEncoding => (define some encodings for byte[], JSON, PDX, *, etc) > key => EncodedValue > > GetResponse => 1 value > 1 => the API id > value => EncodedValue > > PutRequest => 2 eventId key value > 2 => the API id > eventId => clientId threadId sequenceId > clientId => string > threadId => integer > sequenceId => integer > key => EncodedValue > value => EncodedValue > > EncodedValue => encoding (boolean | integer | number | string | ((length > bytes)* 0)) > encoding => (define some encodings for byte[], JSON, PDX, *, etc) > boolean => TRUE or FALSE > integer => a signed integer value > number => a decimal value corresponding to IEEE 754 > string => UTF-8 text > bytes => arbitrary byte[] that can be chunked > > > Anthony
