There would be one Message containing a single MessageHeader and a single MessageBody. A PDX EncodedValue containing 1242 bytes that are chunked would look something like this:
PDX 1000 byte[1000] 242 byte[242] 0 Anthony > On May 4, 2017, at 10:38 AM, Hitesh Khamesra <hitesh...@yahoo.com> wrote: > > Hi Anthony: > > Help me to understand data chunking here? > >>> bytes => arbitrary byte[] that can be chunked > > Message => MessageHeader MessageBody > > So lets say we want to send long byte[] into two chunks, then we will send > two messages? And other side will combine those two messages using > "correlationId" ? > > Thanks. > HItesh > > > > > ________________________________ > From: Anthony Baker <aba...@pivotal.io> > To: dev@geode.apache.org > Cc: Hitesh Khamesra <hitesh...@yahoo.com> > Sent: Wednesday, May 3, 2017 5:42 PM > Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal > > > > >> On May 3, 2017, at 1:33 PM, Galen M O'Sullivan <gosulli...@pivotal.io> wrote: >> >> On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra < >> hitesh...@yahoo.com.invalid> wrote: >> >>> Absolutely its a implementation detail. >>> >> This doesn't answer Dan's comment. Do you think fragmentation should be >> taken care of by the TCP layer or the protocol should deal with it >> specifically? > > There’s some really good feedback and discussion in this thread! Here are a > few thoughts: > > 1) Optional metadata should be used for fields that are generally applicable > across all messages. If a metadata field is required or only applies to a > small set of messages, it should become part of a message definition. Of > course there’s some grey area here. > > 2) I think we should pull out the message fragmentation support to avoid some > significant complexity. We can later add a fragmentation / envelope layer on > top without disrupting the current proposal. I do think we should add the > capability for chunking data (more on that below). > > 3) I did not find any discussion of message pipelining (queuing multiple > requests on a socket without waiting for a response) or out-of-order > responses. What is the plan for these capabilities and how will that affect > consistency? What about retries? > > 4) Following is an alternative definition with these characteristics: > > - Serialized data can either be primitive or encoded values. Encoded values > are chunked as needed to break up large objects into a series of smaller > parts. > - Because values can be chunked, the size field is removed. This allows the > message to be streamed to the socket incrementally. > - The apiVersion is removed because we can just define a new body type with a > new apiId (e.g. GetRequest2 with aipId = 1292). > - The GetRequest tells the server what kind of encoding the client is able to > understand. > - The metadata map is not used for fields that belong in the message body. I > think it’s much easier to write a spec without if statements :-) > > Message => MessageHeader MessageBody > > MessageHeader => correlationId metadata > correlationId => integer > metadata => count (key value)* > count => integer > key => string > value => string > > MessageBody => apiId body > apiId => integer > body => (see specific definitions) > > GetRequest => 0 acceptEncoding key > 0 => the API id > acceptEncoding => (define some encodings for byte[], JSON, PDX, *, etc) > key => EncodedValue > > GetResponse => 1 value > 1 => the API id > value => EncodedValue > > PutRequest => 2 eventId key value > 2 => the API id > eventId => clientId threadId sequenceId > clientId => string > threadId => integer > sequenceId => integer > key => EncodedValue > value => EncodedValue > > EncodedValue => encoding (boolean | integer | number | string | ((length > bytes)* 0)) > encoding => (define some encodings for byte[], JSON, PDX, *, etc) > boolean => TRUE or FALSE > integer => a signed integer value > number => a decimal value corresponding to IEEE 754 > string => UTF-8 text > bytes => arbitrary byte[] that can be chunked > > > Anthony
