On Thu, May 4, 2017 at 2:14 PM, Jacob Barrett wrote:
> > > One benefit of messageHeader with chunk is that, it gives us ability to
> > > write different messages(multiplexing) on same socket. And if thread is
> > > ready it can write its message. Though we are not there yet, but that
> will
> > >
I think we should be presenting the current proposal as the API and message
structure, as would be laid out in an IDL. This way, we can experiment with
Protobuf, Thrift serialization, &c. for message structure without having to
exactly specify the message structure. This will make designing a proto
+1 William, an even better example - that kind of representation will make
it so much better/easier for geode users to implement against, regardless
of language.
On Mon, May 8, 2017 at 2:48 PM, William Markito Oliveira <
william.mark...@gmail.com> wrote:
> +1
>
> I think I've shared this before,
+1
I think I've shared this before, but Kafka also has good (tabular)
representation for messages on their protocol.
- https://kafka.apache.org/protocol#protocol_messages
- https://kafka.apache.org/protocol#protocol_message_sets
On Mon, May 8, 2017 at 4:44 PM, Ernest Burghardt
wrote:
> Hello G
Hello Geodes!
Good discussion on what/how the messages will be/handled once a connection
is established.
+1 to a simple initial handshake to establish version/supported features
that client/server will be communicating.
>From what I've seen so far in the proposal it is missing a definition for
t
On Fri, May 5, 2017 at 2:09 PM Hitesh Khamesra wrote:
>
> 0. In first phase we are not doing chunking/fragmentation. And even this
> will be option for client.(
> https://cwiki.apache.org/confluence/display/GEODE/Message+Structure+and+Definition#MessageStructureandDefinition-Protocolnegotiation
>
chunked input stream and can de-serialize the object as it is
coming (DataSerializable.fromData(ChunkedStream)).
From: Jacob Barrett
To: dev@geode.apache.org; Hitesh Khamesra
Cc: Anthony Baker
Sent: Friday, May 5, 2017 7:29 AM
Subject: Re: [gemfire-dev] New Client-Server Protocol
In either case you packetize (serialize the message protocol) to buffers (fixed
sizes and pooled) and flush buffers to the socket. Preferably using a async
socket framework to do all the heavy lifting for you.
Sent from my iPhone
> On May 5, 2017, at 11:07 AM, Bruce Schuchardt wrote:
>
> Yes
Yes, of course it does but we don't serialize directly to a socket
output stream because it's slow. I agree that this could be left out
and added later as an optimization.
Le 5/5/2017 à 10:33 AM, Galen M O'Sullivan a écrit :
I think TCP does exactly this for us.
On Fri, May 5, 2017 at 9:08 A
It does!
Both fragmenting and multiple channels as multiple sockets.
Sent from my iPhone
> On May 5, 2017, at 10:33 AM, Galen M O'Sullivan wrote:
>
> I think TCP does exactly this for us.
>
> On Fri, May 5, 2017 at 9:08 AM, Bruce Schuchardt
> wrote:
>
>> This is very similar to how peer-t
I think TCP does exactly this for us.
On Fri, May 5, 2017 at 9:08 AM, Bruce Schuchardt
wrote:
> This is very similar to how peer-to-peer messaging is performed in Geode.
> Messages are serialized to a stream that knows how to optimally "chunk" the
> bytes into fixed-size packets. On the receivi
I would leave it for a later optimization.
Sent from my iPhone
> On May 5, 2017, at 9:08 AM, Bruce Schuchardt wrote:
>
> This is very similar to how peer-to-peer messaging is performed in Geode.
> Messages are serialized to a stream that knows how to optimally "chunk" the
> bytes into fixed-
This is very similar to how peer-to-peer messaging is performed in
Geode. Messages are serialized to a stream that knows how to optimally
"chunk" the bytes into fixed-size packets. On the receiving side these
are fed into a similar input stream for deserialization. The message
only contains
On Thu, May 4, 2017 at 2:52 PM Hitesh Khamesra
wrote:
> Basically, thread/layer should not hold any resources while serializing
> the object or chunk. We should be able to see this flow (ms1-chunk1,
> msg2-chunk1, ms1-chunk2, msg3-chunk, msg2-chunk2, so on ...)
>
Correct, but putting that in th
+1 for that as well
On Thu, May 4, 2017 at 5:21 PM Dan Smith wrote:
> >
> > I wouldn't tackle that at this layer. I would suggest adding a layer
> > between the message and TCP that creates channels that are opaque to the
> > message layer above. The message layer wouldn't know if it was talking
>
> I wouldn't tackle that at this layer. I would suggest adding a layer
> between the message and TCP that creates channels that are opaque to the
> message layer above. The message layer wouldn't know if it was talking to
> multiple sockets to the client or single socket with multiple channels.
>
, May 4, 2017 2:14 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
> One benefit of messageHeader with chunk is that, it gives us ability to
> write different messages(multiplexing) on same socket. And if thread is
> ready it can write its message. Though we are not
> One benefit of messageHeader with chunk is that, it gives us ability to
> write different messages(multiplexing) on same socket. And if thread is
> ready it can write its message. Though we are not there yet, but that will
> require for single socket async architecture.
>
I wouldn't tackle that
ite its message. Though we are not there yet, but that will require for
single socket async architecture.
From: Jacob Barrett
To: dev@geode.apache.org
Cc: Anthony Baker
Sent: Thursday, May 4, 2017 12:48 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Pro
Sent from my iPhone
> On May 4, 2017, at 12:03 PM, Hitesh Khamesra
> wrote:
>
> And len 0 would indicate end of the message?
>
>
> a. Now these two chunks will go continuous.
They would appear continuous to the object serialization layer.
>
>
> b. If its PDX encoded then pdx header(1
tand this better.
Thanks.
Hitesh
From: Anthony Baker
To: Hitesh Khamesra
Cc: "dev@geode.apache.org"
Sent: Thursday, May 4, 2017 11:20 AM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
There would be one Message containing a single Me
other side will combine those two messages using
> "correlationId" ?
>
> Thanks.
> HItesh
>
>
>
>
>
> From: Anthony Baker
> To: dev@geode.apache.org
> Cc: Hitesh Khamesra
> Sent: Wednesday, May 3, 2017 5:42 PM
>
ssages using "correlationId" ?
Thanks.
HItesh
From: Anthony Baker
To: dev@geode.apache.org
Cc: Hitesh Khamesra
Sent: Wednesday, May 3, 2017 5:42 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
> On May 3, 2017, at 1:33 PM, Galen M O
It's becoming clear that the document needs a section on EventIds.
EventIds aren't opaque to the server. They are comparable objects and
are used by the cache to prevent replay of older (operation eventId <
recorded eventId) operations on the cache and on subscription queues.
They are also u
+1
On Wed, May 3, 2017 at 5:43 PM Anthony Baker wrote:
>
> 2) I think we should pull out the message fragmentation support to avoid
> some significant complexity. We can later add a fragmentation / envelope
> layer on top without disrupting the current proposal. I do think we should
> add the
+1 to what Anthony has laid out! I think this is a better way to handle
value encodings, and it's also better to be putting message specific
details like event id with those messages.
I do wonder whether this proposal actually needs metadata headers at all?
What will eventually go in there?
-Dan
Good point Dan !! that needs to document.
From: Dan Smith
To: dev@geode.apache.org
Sent: Wednesday, May 3, 2017 5:31 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
Okay but how do I has an implementer of a driver know what messages
need an event id and
> On May 3, 2017, at 1:33 PM, Galen M O'Sullivan wrote:
>
> On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra <
> hitesh...@yahoo.com.invalid> wrote:
>
>> Absolutely its a implementation detail.
>>
> This doesn't answer Dan's comment. Do you think fragmentation should be
> taken care of by the
Okay but how do I has an implementer of a driver know what messages
need an event id and which don't? It seems like maybe this belongs with
those message types, rather than in a generic header. Or maybe you need to
start organizing messages into classes - eg messages that change state and
mess
Correct,
I did miss that. @Dan, if you look at
https://cwiki.apache.org/confluence/display/GEODE/Message+Structure+and+Definition#MessageStructureandDefinition-MetaDataforRequests
specifies how we provide EventId information.
On 5/3/17 09:53, Bruce Schuchardt wrote:
I believe Hitesh put Eve
: Wednesday, May 3, 2017 10:01 AM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
Here are the few things we need to consider..
1. key, value, callbackarg can be required to interpret as JSON-to-pdx2. client calls
"get/getall" api and want return value as JSON. Value
The TCP fragmentation is fine for what it is, but it is *not* paging, and
paging has long been something that we have wanted to get around to.
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771
On Wed, May 3, 2017 at 1:33 PM, Galen M O'Sullivan
wrote:
> On Tue, Ma
On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra <
hitesh...@yahoo.com.invalid> wrote:
> Absolutely its a implementation detail.
>
This doesn't answer Dan's comment. Do you think fragmentation should be
taken care of by the TCP layer or the protocol should deal with it
specifically?
o json or vice-versa.
Any thoughts?
Thanks.
HItesh
From: Hitesh Khamesra
To: "dev@geode.apache.org"
Sent: Wednesday, May 3, 2017 10:01 AM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
Here are the few things we need to consider..
From: Jacob Barrett
To: dev@geode.apache.org
Cc: Udo Kohlmeyer
Sent: Tuesday, May 2, 2017 8:11 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
I agree completely with Dan. There is no reason to have flags for value
encoding type in the message. I would argue that s
I believe Hitesh put EventId in the metadata section.
Le 5/2/2017 à 2:22 PM, Udo Kohlmeyer a écrit :
We are considering the function service, but again, this should not
detract from the proposed message specification proposal.
You are also correct in your observation of list of error codes not
From: Michael Stolz
To: dev@geode.apache.org
Cc: Udo Kohlmeyer
Sent: Wednesday, May 3, 2017 8:55 AM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
I'm not seeing any mention of versioning of the serialization protocol.
Versioning is critical to be able to support c
I'm not seeing any mention of versioning of the serialization protocol.
Versioning is critical to be able to support change over time. We must
version each layer of serialization. The transport message needs versions,
the payload serialization needs versions.
--
Mike Stolz
Principal Engineer, GemF
I agree completely with Dan. There is no reason to have flags for value
encoding type in the message. I would argue that should be part of the value
serialization layer. If something was placed in the message layer it should be
more generic and allow for an unrestricted set of encodings by some
We are considering the function service, but again, this should not
detract from the proposed message specification proposal.
You are also correct in your observation of list of error codes not
being complete nor exhaustive. Maybe the first page needs to highlight
that this is a proposal and d
I guess the value of building other messages on top of the function service
mostly comes into play when we start talking about smarter clients that can
do single hop. At that point it's really nice to have have a layer that
lets us send a message to a single primary, or all of the members that host
: [gemfire-dev] New Client-Server Protocol Proposal
> IsPartialMessage: This flag gives us ability to send partial message
> without serializing the whole key-value(request). lets say I execute
> function at server and function just returns "arraylist of object". And the
> serializ
> IsPartialMessage: This flag gives us ability to send partial message
> without serializing the whole key-value(request). lets say I execute
> function at server and function just returns "arraylist of object". And the
> serialized size of ""arraylist of object"" is quite big( > 2gb).
>
My point
Hey Dan,
Imo, having a standardized, versioned definition for GET, PUT, PUTALL,
etc. message, that is encoded/decoded in a manner that multiple clients
(written in many other languages) can encode/decode these messages, is
paramount.
Having the standardized operational messages(GET,PUT,etc.)
I think the downside of having a single generic message type is that we lose
“type safety” and some efficiency. The message definition would essentially
become:
String functionName;
byte[][] args;
It’s a little more challenging for an author of a Geode Driver to fill in the
args correctly com
t can send JSON string and we want to save that
JSON string into pdx.
From: Dan Smith
To: Udo Kohlmeyer
Cc: dev@geode.apache.org
Sent: Monday, May 1, 2017 5:53 PM
Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal
I think the current proposal seems to be gloming so
I think the current proposal seems to be gloming some things together that
probably belong in different layers.
The message header currently is specified to have things like correlation
id, isPartial message, and also metdatadata about whether the key or the
value is JSON. Fragmenting messages (is
I think any new client driver or server we develop might want to
incorporate function execution at lower level than region operations like
get and put, etc. We could then easily build operations like GET, PUT,
PUTALL, etc. on top of that by making them functions. The original client
protocol isn't
48 matches
Mail list logo
