Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael William Dodge]

I think there is value in having a single string encoding.

Sarge

> On 1 Dec, 2017, at 17:35, Jacob Barrett  wrote:
> 
> On Fri, Dec 1, 2017 at 4:59 PM Dan Smith  wrote:
> 
>> I think I'm kinda with Mike on this one. The existing string format does
>> seem pretty gnarly. But the complexity of implementing and testing all of
>> the backwards compatibility transcoding that would be required in order to
>> move to the new proposed format seems to be way more work with much more
>> possibility for errors. Do we really expect people to be writing new
>> clients that use DataSerializable? It hasn't happened yet, and we're
>> working on a new protocol that uses protobuf right now.
>> 
> 
> Consider that any new clients written would have to implement all these
> encodings. This is going to make writing new clients using the upcoming new
> protocol laborious. The new protocol does not define object encoding, it
> strictly defines message encoding. Objects sent over the protocol will have
> to be serialized in some format, like PDX or data serializer. We could
> alway develop a better serialization format than what we have now. If we
> don't develop something new then we have to use the old. Wouldn't it be
> nice if the new clients didn't have to deal with legacy encodings?
> 
> If the issue is really the complexity of serialization from the C++ client,
>> maybe the C++ client could always write UTF-16 strings?
>> 
> 
> You can't assume that a client in one language will only be serializing
> strings for it's own consumption. We have many people using strings in PDX
> to transform between C++, .NET and Java.
> 
> The risk is high not to remove this debt. If I am developing a new Ruby
> client I am forced to deal with all 4 of these encodings. Am I really going
> to want to build a Ruby client for Geode, am I going to get these encodings
> correct? I can tell you that getting them correct may be a challenge if the
> current C++ client is any indication, it has a few incorrect assumptions in
> its encoding of ASCII and modified UTF-8.
> 
> I am fine with a compromise that deprecates but doesn't remove the old
> encodings for a few releases. This would give time for users to update. New
> clients written would not be be able to read this old data but could read
> and write new data.
> 
> 
> 
> -Jake

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Dan Smith]

The new protocol is currently translating from PDX->JSON before sending
results to the clients so the client doesn't have to understand PDX or
DataSerializable.

There is a lot more to DataSerializable than just how a String is
serialized. And it's not even documented that I am aware of. Just tweaking
the string format is not going to make that much better. Your hypothetical
ruby developer is in trouble with or without this proposed change.

Breaking compatibility is a huge PITA for our users. We should do that when
we are actually giving them real benefits. In this case if we were
switching to some newer PDX format that was actually easy to implement
deserialization logic I could see the argument for breaking compatibility.
Just changing the string format without fixing the rest of the issues
around DataSerializable isn't providing real benefits.

You can't assume that a client in one language will only be serializing
> strings for it's own consumption.
>

I wasn't making that assumption. The suggestion is that the C++ client
would have to deserialize all 4 valid formats, but it could just always
serialize data using the valid UTF-16 format. All other clients should be
able to deserialize that.

-Dan

On Fri, Dec 1, 2017 at 5:35 PM, Jacob Barrett  wrote:

> On Fri, Dec 1, 2017 at 4:59 PM Dan Smith  wrote:
>
> > I think I'm kinda with Mike on this one. The existing string format does
> > seem pretty gnarly. But the complexity of implementing and testing all of
> > the backwards compatibility transcoding that would be required in order
> to
> > move to the new proposed format seems to be way more work with much more
> > possibility for errors. Do we really expect people to be writing new
> > clients that use DataSerializable? It hasn't happened yet, and we're
> > working on a new protocol that uses protobuf right now.
> >
>
> Consider that any new clients written would have to implement all these
> encodings. This is going to make writing new clients using the upcoming new
> protocol laborious. The new protocol does not define object encoding, it
> strictly defines message encoding. Objects sent over the protocol will have
> to be serialized in some format, like PDX or data serializer. We could
> alway develop a better serialization format than what we have now. If we
> don't develop something new then we have to use the old. Wouldn't it be
> nice if the new clients didn't have to deal with legacy encodings?
>
> If the issue is really the complexity of serialization from the C++ client,
> > maybe the C++ client could always write UTF-16 strings?
> >
>
> You can't assume that a client in one language will only be serializing
> strings for it's own consumption. We have many people using strings in PDX
> to transform between C++, .NET and Java.
>
> The risk is high not to remove this debt. If I am developing a new Ruby
> client I am forced to deal with all 4 of these encodings. Am I really going
> to want to build a Ruby client for Geode, am I going to get these encodings
> correct? I can tell you that getting them correct may be a challenge if the
> current C++ client is any indication, it has a few incorrect assumptions in
> its encoding of ASCII and modified UTF-8.
>
> I am fine with a compromise that deprecates but doesn't remove the old
> encodings for a few releases. This would give time for users to update. New
> clients written would not be be able to read this old data but could read
> and write new data.
>
>
>
> -Jake
>

[Spring CI] Spring Data GemFire > Nightly-ApacheGeode > #756 was SUCCESSFUL (with 2264 tests)

[Spring CI]

---
Spring Data GemFire > Nightly-ApacheGeode > #756 was successful.
---
Scheduled
2266 tests in total.

https://build.spring.io/browse/SGF-NAG-756/





--
This message is automatically generated by Atlassian Bamboo

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

Anything that breaks data on disk is also a big PITA. This change would
break data on disk.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Dec 4, 2017 at 1:52 PM, Dan Smith  wrote:

> The new protocol is currently translating from PDX->JSON before sending
> results to the clients so the client doesn't have to understand PDX or
> DataSerializable.
>
> There is a lot more to DataSerializable than just how a String is
> serialized. And it's not even documented that I am aware of. Just tweaking
> the string format is not going to make that much better. Your hypothetical
> ruby developer is in trouble with or without this proposed change.
>
> Breaking compatibility is a huge PITA for our users. We should do that when
> we are actually giving them real benefits. In this case if we were
> switching to some newer PDX format that was actually easy to implement
> deserialization logic I could see the argument for breaking compatibility.
> Just changing the string format without fixing the rest of the issues
> around DataSerializable isn't providing real benefits.
>
> You can't assume that a client in one language will only be serializing
> > strings for it's own consumption.
> >
>
> I wasn't making that assumption. The suggestion is that the C++ client
> would have to deserialize all 4 valid formats, but it could just always
> serialize data using the valid UTF-16 format. All other clients should be
> able to deserialize that.
>
> -Dan
>
> On Fri, Dec 1, 2017 at 5:35 PM, Jacob Barrett  wrote:
>
> > On Fri, Dec 1, 2017 at 4:59 PM Dan Smith  wrote:
> >
> > > I think I'm kinda with Mike on this one. The existing string format
> does
> > > seem pretty gnarly. But the complexity of implementing and testing all
> of
> > > the backwards compatibility transcoding that would be required in order
> > to
> > > move to the new proposed format seems to be way more work with much
> more
> > > possibility for errors. Do we really expect people to be writing new
> > > clients that use DataSerializable? It hasn't happened yet, and we're
> > > working on a new protocol that uses protobuf right now.
> > >
> >
> > Consider that any new clients written would have to implement all these
> > encodings. This is going to make writing new clients using the upcoming
> new
> > protocol laborious. The new protocol does not define object encoding, it
> > strictly defines message encoding. Objects sent over the protocol will
> have
> > to be serialized in some format, like PDX or data serializer. We could
> > alway develop a better serialization format than what we have now. If we
> > don't develop something new then we have to use the old. Wouldn't it be
> > nice if the new clients didn't have to deal with legacy encodings?
> >
> > If the issue is really the complexity of serialization from the C++
> client,
> > > maybe the C++ client could always write UTF-16 strings?
> > >
> >
> > You can't assume that a client in one language will only be serializing
> > strings for it's own consumption. We have many people using strings in
> PDX
> > to transform between C++, .NET and Java.
> >
> > The risk is high not to remove this debt. If I am developing a new Ruby
> > client I am forced to deal with all 4 of these encodings. Am I really
> going
> > to want to build a Ruby client for Geode, am I going to get these
> encodings
> > correct? I can tell you that getting them correct may be a challenge if
> the
> > current C++ client is any indication, it has a few incorrect assumptions
> in
> > its encoding of ASCII and modified UTF-8.
> >
> > I am fine with a compromise that deprecates but doesn't remove the old
> > encodings for a few releases. This would give time for users to update.
> New
> > clients written would not be be able to read this old data but could read
> > and write new data.
> >
> >
> >
> > -Jake
> >
>

Re: [PDE East] Geode Conference - Double the Expected Attendance

[Michael Stolz]

This was really a great day in the history of Apache Geode.

We were concerned that we might not have any turn-out at all for this
session because it was disjointed from the rest of Spring 1 Platform. The
rest of the show starts tomorrow.

We were concerned that we had no way to measure the popularity because
there was no registration page.
We knew we had way too many great speaker submissions to turn down all but
7 of them in the main show starting Tuesday.
In fact we still had to turn down quite a few (including mine :)


Thank goodness the Spring 1 app has an "add to my schedule" feature.


We got wind a few days ago that there were 80+ "add to my schedule" hits.
The room was only configured to handle 65 people.

Jagdish worked with the folks running the conference to reconfigure the
room to accommodate about 170 people by changing the layout.

We filled all 170 seats 30 minutes before the session started. We
proactively moved briefcases off of seats, moved people to the sides of the
room and made every seat count. We still couldn't fit everybody.

I want to thank all the speakers, the PDE team, the event team and Jagdish
for all of the efforts that went into making this the best attended Geode
event ever. Clearly Geode is up and coming. More than double last year's
attendance. And we had to turn some away.

Let's start looking at how we want to continue to grow the community next
year. And lets not wait until December. We have lots of opportunity to grow
the Geode community in the early part of 2018.

Thanks all for the great work leading up to this fantastic outcome.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Dec 4, 2017 at 10:01 PM, Jagdish Mirani  wrote:

> Yes, we promoted the book. As Wes noted, we totally exceeded our
> expectations. ☺
>
> On Mon, Dec 4, 2017 at 1:33 PM, Marshall Presser 
> wrote:
>
>> Please promote Mike's book if appropriate.
>> MEP
>>
>> On Mon, Dec 4, 2017 at 4:16 PM, Wes Williams 
>> wrote:
>>
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>> Wes Williams
>>> Sent from mobile phone
>>>
>>
>>
>>
>> --
>> Marshall Presser
>> Pivotal Data Engineering
>> mpresser@pivotal .io
>> 240.401.1750 <(240)%20401-1750>
>>
>>
>
>
> --
> Regards
> Jag
>

Re: [PDE East] Geode Conference - Double the Expected Attendance

[Akihiro Kitada]

Great news!


-- 
Akihiro Kitada  |  Staff Customer Engineer |  +81 80 3716 3736
Support.Pivotal.io   |  Mon-Fri  9:00am to
5:30pm JST  |  1-877-477-2269
[image: support]  [image: twitter]
 [image: linkedin]
 [image: facebook]
 [image: google plus]
 [image: youtube]



2017-12-05 16:01 GMT+09:00 Michael Stolz :

> This was really a great day in the history of Apache Geode.
>
> We were concerned that we might not have any turn-out at all for this
> session because it was disjointed from the rest of Spring 1 Platform. The
> rest of the show starts tomorrow.
>
> We were concerned that we had no way to measure the popularity because
> there was no registration page.
> We knew we had way too many great speaker submissions to turn down all but
> 7 of them in the main show starting Tuesday.
> In fact we still had to turn down quite a few (including mine :)
>
>
> Thank goodness the Spring 1 app has an "add to my schedule" feature.
>
>
> We got wind a few days ago that there were 80+ "add to my schedule" hits.
> The room was only configured to handle 65 people.
>
> Jagdish worked with the folks running the conference to reconfigure the
> room to accommodate about 170 people by changing the layout.
>
> We filled all 170 seats 30 minutes before the session started. We
> proactively moved briefcases off of seats, moved people to the sides of the
> room and made every seat count. We still couldn't fit everybody.
>
> I want to thank all the speakers, the PDE team, the event team and Jagdish
> for all of the efforts that went into making this the best attended Geode
> event ever. Clearly Geode is up and coming. More than double last year's
> attendance. And we had to turn some away.
>
> Let's start looking at how we want to continue to grow the community next
> year. And lets not wait until December. We have lots of opportunity to grow
> the Geode community in the early part of 2018.
>
> Thanks all for the great work leading up to this fantastic outcome.
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771 <+1%20631-835-4771>
>
> On Mon, Dec 4, 2017 at 10:01 PM, Jagdish Mirani 
> wrote:
>
>> Yes, we promoted the book. As Wes noted, we totally exceeded our
>> expectations. ☺
>>
>> On Mon, Dec 4, 2017 at 1:33 PM, Marshall Presser 
>> wrote:
>>
>>> Please promote Mike's book if appropriate.
>>> MEP
>>>
>>> On Mon, Dec 4, 2017 at 4:16 PM, Wes Williams 
>>> wrote:
>>>





 Regards,
 Wes Williams
 Sent from mobile phone

>>>
>>>
>>>
>>> --
>>> Marshall Presser
>>> Pivotal Data Engineering
>>> mpresser@pivotal .io
>>> 240.401.1750 <(240)%20401-1750>
>>>
>>>
>>
>>
>> --
>> Regards
>> Jag
>>
>
>