Re: About operating system statistics

[Alberto Bustamante Reyes]

pull request available: https://github.com/apache/geode/pull/3574

[https://avatars3.githubusercontent.com/u/47359?s=400&v=4]

GEODE-6660: Remove non-Linux OS statistics classes by alb3rtobr · Pull Request 
#3574 · apache/geode
github.com
Thank you for submitting a contribution to Apache Geode. In order to streamline 
the review of the contribution we ask you to ensure the following steps have 
been taken: For all changes: Is there...




De: Alberto Bustamante Reyes 
Enviado: martes, 16 de abril de 2019 15:21:34
Para: dev@geode.apache.org
Asunto: RE: About operating system statistics

Thanks for your answer Darrel. I have created a jira to delete these classes.

BR/

Alberto

De: Darrel Schneider 
Enviado: lunes, 15 de abril de 2019 17:12
Para: dev@geode.apache.org
Asunto: Re: About operating system statistics

The code you have found predates the open source geode project. That code
requires a native, platform dependent, jar to be distributed. We decided
that geode would be pure java. The code that collects Linux stats is pure
java. So I think the code you have found for Solaris, Windows, and OSX has
been "abandoned" and can be deleted.


On Mon, Apr 15, 2019 at 6:54 AM Alberto Bustamante Reyes
 wrote:

> Hi all,
>
> I have read in the documentation that the operating system statistics are
> only available for Linux systems, but I have found in the code the
> correspondent classes for OSX, Solaris & Windows stats (package
> org.apache.geode.internal.statistics.platform).
>
> Are these classes being used? Should they be included in the documentation
> or deleted? (the OSX ones are almost empty, so Im not sure if someone is
> working on it or its an 'abandonded feature')
>
> Thanks in advance,
>
> BR/
>
> Alberto
>

Re: Geode self-protection about overload

[Anthony Baker]

The primary load limiter between the client tier and the Geode servers is via 
the max connections limit as noted in this writeup:

https://cwiki.apache.org/confluence/display/GEODE/Resource+Management+in+Geode 


When the load is sufficiently high, operations may timeout and a geode client 
will failover to less loaded servers.  You can limit the number of retries the 
client will attempt (each gated by a read timeout) and thus slow down incoming 
operations.  

We’re looking into some improvements in the client connection pool to improve 
both performance and behaviors at the ragged edge when resources are saturated. 
 Contributions welcome!

Anthony


> On May 13, 2019, at 9:02 AM, Alberto Gomez  wrote:
> 
> Hi Anthony!
> 
> Thanks a lot for your prompt answer.
> 
> I think it is great that Geode can preserve the availability and predictable 
> low latency of the cluster when some members are unresponsive by means of the 
> GMS.
> 
> My question was more targeted to situations in which the load received by the 
> cluster is so high that all members struggle to offer low latency. Under such 
> circumstances, does Geode take any action to back-off some of the incoming 
> load?
> 
> Thanks in advance,
> 
> Alberto
> 
> 
> On 10/5/19 17:52, Anthony Baker wrote:
> 
> Hi Alberto!
> 
> Great questions.  One of the fundamental characteristics of Geode is its 
> Group Membership System (GMS).  You can read more about it here [1].  The 
> membership system ensures that failures due to unresponsive members and/or 
> network partitions are detected quickly.  Given that we use synchronous 
> replication for consistent updates, the GMS algorithms fence off unresponsive 
> members to preserve the availability (and predictable low latency) of the 
> cluster as a whole.
> 
> Another factor of resilience is memory load.  Regions can be configured to 
> automatically evict data to disk based on heap usage.  In addition, when a 
> Region exceeds a critical memory usage thresholds further updates are blocked 
> until the overload is resolved.
> 
> Geode clients route operations to cluster members based on connection load.  
> This helps balance cpu load across the entire cluster.  Cluster members can 
> set connection maximums to prevent overrunning the available capacity of an 
> individual server.
> 
> I hope this helps and feel free to keep asking questions :-)
> 
> Anthony
> 
> [1] 
> https://cwiki.apache.org/confluence/display/GEODE/Core+Distributed+System+Concepts
>  
>   
> >
> 
> 
> 
> 
> On May 10, 2019, at 3:22 AM, Alberto Gomez  wrote:
> 
> Hi Geode community!
> 
> I'd like to know if Geode implements any kind of self-protection against 
> overload. What I mean by this is some mechanism that allows Geode servers 
> (and possibly locators) to reject incoming operations before processing them 
> when it detects that it is not able to handle the amount of operations 
> received in a reasonable way (with reasonable latency and without 
> experiencing processes crashing).
> 
> The goal would be to make sure that Geode (or some parts of it) do not crash 
> under too heavy load and also that the latency level is always under control 
> at least for the amount of traffic the Geode cluster is supposed to support.
> 
> If Geode does not offer such mechanism, I would also like to get your opinion 
> about this possible feature, (if you find it interesting) and also on how it 
> could be implemented. One possible approach could be having some measure of 
> the current CPU consumption that allows to decide if a given operation must 
> be processed or not, taking into account the CPU consumption value with 
> respect to an overload threshold.
> 
> Thanks in advance for your answers,
> 
> -Alberto

Backwards compatibility issue with JSONFormatter

[Dan Smith]

I think the changes for GEODE-6327 broke backwards compatibility in
JSONFormatter with the change from fromJSON(String jsonString) to
fromJSON(String jsonString, String... identityFields)

Adding an additional varargs parameter to the method breaks code that was
compiled against the non-varargs version. I think we need to overload the
method instead.

Thanks to Gester for discovering this with his test!

-Dan

Re: Backwards compatibility issue with JSONFormatter

[Udo Kohlmeyer]

How did this slip the review process that the signature of a public API 
was changed?


Well done Gester for finding this!!

--Udo

On 5/14/19 10:40, Dan Smith wrote:

I think the changes for GEODE-6327 broke backwards compatibility in
JSONFormatter with the change from fromJSON(String jsonString) to
fromJSON(String jsonString, String... identityFields)

Adding an additional varargs parameter to the method breaks code that was
compiled against the non-varargs version. I think we need to overload the
method instead.

Thanks to Gester for discovering this with his test!

-Dan

Re: Backwards compatibility issue with JSONFormatter

[Ryan McMahon]

Hi all,

This is my mistake - it was a misunderstanding of what constitutes a
breaking public API change.  If a client app were to recompile using the
new bits with the new method signature, there wouldn't be an issue because
the new signature would work with 0 varargs.  The problem arises when you
hot swap the geode dependencies for that client app without recompiling, as
the bytecode no longer matches.  Since we do support the hot swap use case
for CVEs etc, I see now this is considered a breaking API change.

I'll change this to use a method overload instead, it should be a pretty
simple fix.  Luckily, this issue didn't make it into any Geode releases.

Thanks,
Ryan


On Tue, May 14, 2019 at 10:47 AM Udo Kohlmeyer  wrote:

> How did this slip the review process that the signature of a public API
> was changed?
>
> Well done Gester for finding this!!
>
> --Udo
>
> On 5/14/19 10:40, Dan Smith wrote:
> > I think the changes for GEODE-6327 broke backwards compatibility in
> > JSONFormatter with the change from fromJSON(String jsonString) to
> > fromJSON(String jsonString, String... identityFields)
> >
> > Adding an additional varargs parameter to the method breaks code that was
> > compiled against the non-varargs version. I think we need to overload the
> > method instead.
> >
> > Thanks to Gester for discovering this with his test!
> >
> > -Dan
> >
>

Re: Permission for working on Jira

[Geet Gmail]

Now it’s working. Thank you Dan.


Sent from my iPhone

> On May 13, 2019, at 5:44 PM, Dan Smith  wrote:
> 
> Hi Geet,
> 
> Welcome! I think I already gave you access to work with geode JIRAs earlier? 
> I see you listed as having access in JIRA. Is there something you are trying 
> to do that isn't working?
> 
> -Dan
> 
>> On Mon, May 13, 2019 at 2:23 PM Geetanjali rawat  
>> wrote:
>> Hello Devs
>> 
>> I would like to request for the permission for working on Geode Jira.
>> 
>> Thanks
>> Geet

Re: Backwards compatibility issue with JSONFormatter

[Dan Smith]

Sounds good! Yeah, this is a bit of an interesting case where code will
still compile against the new API without change, but I think maintaining
compatibility of the binary API is also important. Thanks for looking into
it.

-Dan

On Tue, May 14, 2019 at 11:22 AM Ryan McMahon  wrote:

> Hi all,
>
> This is my mistake - it was a misunderstanding of what constitutes a
> breaking public API change.  If a client app were to recompile using the
> new bits with the new method signature, there wouldn't be an issue because
> the new signature would work with 0 varargs.  The problem arises when you
> hot swap the geode dependencies for that client app without recompiling, as
> the bytecode no longer matches.  Since we do support the hot swap use case
> for CVEs etc, I see now this is considered a breaking API change.
>
> I'll change this to use a method overload instead, it should be a pretty
> simple fix.  Luckily, this issue didn't make it into any Geode releases.
>
> Thanks,
> Ryan
>
>
> On Tue, May 14, 2019 at 10:47 AM Udo Kohlmeyer  wrote:
>
> > How did this slip the review process that the signature of a public API
> > was changed?
> >
> > Well done Gester for finding this!!
> >
> > --Udo
> >
> > On 5/14/19 10:40, Dan Smith wrote:
> > > I think the changes for GEODE-6327 broke backwards compatibility in
> > > JSONFormatter with the change from fromJSON(String jsonString) to
> > > fromJSON(String jsonString, String... identityFields)
> > >
> > > Adding an additional varargs parameter to the method breaks code that
> was
> > > compiled against the non-varargs version. I think we need to overload
> the
> > > method instead.
> > >
> > > Thanks to Gester for discovering this with his test!
> > >
> > > -Dan
> > >
> >
>

Re: Backwards compatibility issue with JSONFormatter

[Anthony Baker]

Here are a few links on API compatibility:

https://lvc.github.io/japi-compliance-checker/#Examples
https://wiki.eclipse.org/Evolving_Java-based_APIs
https://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html


Anthony


> On May 14, 2019, at 12:45 PM, Dan Smith  wrote:
> 
> Sounds good! Yeah, this is a bit of an interesting case where code will
> still compile against the new API without change, but I think maintaining
> compatibility of the binary API is also important. Thanks for looking into
> it.
> 
> -Dan
> 
> On Tue, May 14, 2019 at 11:22 AM Ryan McMahon  wrote:
> 
>> Hi all,
>> 
>> This is my mistake - it was a misunderstanding of what constitutes a
>> breaking public API change.  If a client app were to recompile using the
>> new bits with the new method signature, there wouldn't be an issue because
>> the new signature would work with 0 varargs.  The problem arises when you
>> hot swap the geode dependencies for that client app without recompiling, as
>> the bytecode no longer matches.  Since we do support the hot swap use case
>> for CVEs etc, I see now this is considered a breaking API change.
>> 
>> I'll change this to use a method overload instead, it should be a pretty
>> simple fix.  Luckily, this issue didn't make it into any Geode releases.
>> 
>> Thanks,
>> Ryan
>> 
>> 
>> On Tue, May 14, 2019 at 10:47 AM Udo Kohlmeyer  wrote:
>> 
>>> How did this slip the review process that the signature of a public API
>>> was changed?
>>> 
>>> Well done Gester for finding this!!
>>> 
>>> --Udo
>>> 
>>> On 5/14/19 10:40, Dan Smith wrote:
 I think the changes for GEODE-6327 broke backwards compatibility in
 JSONFormatter with the change from fromJSON(String jsonString) to
 fromJSON(String jsonString, String... identityFields)
 
 Adding an additional varargs parameter to the method breaks code that
>> was
 compiled against the non-varargs version. I think we need to overload
>> the
 method instead.
 
 Thanks to Gester for discovering this with his test!
 
 -Dan
 
>>> 
>> 

Re: Backwards compatibility issue with JSONFormatter

[Ryan McMahon]

Thanks Anthony!  Have we considered using a compliance checker in our CI
like the one in the first link?

Side note - I think that varargs is an interesting case that I don't see
called out in those links.  Since varargs is just syntactic sugar, it is
ultimately the varargs parameter is converted to an array in the bytecode.
So in the JSON formatter change:

fromJSON(byte[] jsonByteArray, String... identityFields)


becomes

fromJSON(byte[] jsonByteArray, String[] identityFields)


Hence the breakage of the ABI.

Ryan

On Tue, May 14, 2019 at 1:33 PM Anthony Baker  wrote:

> Here are a few links on API compatibility:
>
> https://lvc.github.io/japi-compliance-checker/#Examples
> https://wiki.eclipse.org/Evolving_Java-based_APIs
> https://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html
>
>
> Anthony
>
>
> > On May 14, 2019, at 12:45 PM, Dan Smith  wrote:
> >
> > Sounds good! Yeah, this is a bit of an interesting case where code will
> > still compile against the new API without change, but I think maintaining
> > compatibility of the binary API is also important. Thanks for looking
> into
> > it.
> >
> > -Dan
> >
> > On Tue, May 14, 2019 at 11:22 AM Ryan McMahon 
> wrote:
> >
> >> Hi all,
> >>
> >> This is my mistake - it was a misunderstanding of what constitutes a
> >> breaking public API change.  If a client app were to recompile using the
> >> new bits with the new method signature, there wouldn't be an issue
> because
> >> the new signature would work with 0 varargs.  The problem arises when
> you
> >> hot swap the geode dependencies for that client app without
> recompiling, as
> >> the bytecode no longer matches.  Since we do support the hot swap use
> case
> >> for CVEs etc, I see now this is considered a breaking API change.
> >>
> >> I'll change this to use a method overload instead, it should be a pretty
> >> simple fix.  Luckily, this issue didn't make it into any Geode releases.
> >>
> >> Thanks,
> >> Ryan
> >>
> >>
> >> On Tue, May 14, 2019 at 10:47 AM Udo Kohlmeyer  wrote:
> >>
> >>> How did this slip the review process that the signature of a public API
> >>> was changed?
> >>>
> >>> Well done Gester for finding this!!
> >>>
> >>> --Udo
> >>>
> >>> On 5/14/19 10:40, Dan Smith wrote:
>  I think the changes for GEODE-6327 broke backwards compatibility in
>  JSONFormatter with the change from fromJSON(String jsonString) to
>  fromJSON(String jsonString, String... identityFields)
> 
>  Adding an additional varargs parameter to the method breaks code that
> >> was
>  compiled against the non-varargs version. I think we need to overload
> >> the
>  method instead.
> 
>  Thanks to Gester for discovering this with his test!
> 
>  -Dan
> 
> >>>
> >>
>
>

Re: Backwards compatibility issue with JSONFormatter

[Ryan McMahon]

Fixed in
https://github.com/apache/geode/commit/cc0b37a504480f554b1884491f44a3cca4113ef5

On Tue, May 14, 2019 at 2:06 PM Ryan McMahon  wrote:

> Thanks Anthony!  Have we considered using a compliance checker in our CI
> like the one in the first link?
>
> Side note - I think that varargs is an interesting case that I don't see
> called out in those links.  Since varargs is just syntactic sugar, it is
> ultimately the varargs parameter is converted to an array in the bytecode.
> So in the JSON formatter change:
>
> fromJSON(byte[] jsonByteArray, String... identityFields)
>
>
> becomes
>
> fromJSON(byte[] jsonByteArray, String[] identityFields)
>
>
> Hence the breakage of the ABI.
>
> Ryan
>
> On Tue, May 14, 2019 at 1:33 PM Anthony Baker  wrote:
>
>> Here are a few links on API compatibility:
>>
>> https://lvc.github.io/japi-compliance-checker/#Examples
>> https://wiki.eclipse.org/Evolving_Java-based_APIs
>> https://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html
>>
>>
>> Anthony
>>
>>
>> > On May 14, 2019, at 12:45 PM, Dan Smith  wrote:
>> >
>> > Sounds good! Yeah, this is a bit of an interesting case where code will
>> > still compile against the new API without change, but I think
>> maintaining
>> > compatibility of the binary API is also important. Thanks for looking
>> into
>> > it.
>> >
>> > -Dan
>> >
>> > On Tue, May 14, 2019 at 11:22 AM Ryan McMahon 
>> wrote:
>> >
>> >> Hi all,
>> >>
>> >> This is my mistake - it was a misunderstanding of what constitutes a
>> >> breaking public API change.  If a client app were to recompile using
>> the
>> >> new bits with the new method signature, there wouldn't be an issue
>> because
>> >> the new signature would work with 0 varargs.  The problem arises when
>> you
>> >> hot swap the geode dependencies for that client app without
>> recompiling, as
>> >> the bytecode no longer matches.  Since we do support the hot swap use
>> case
>> >> for CVEs etc, I see now this is considered a breaking API change.
>> >>
>> >> I'll change this to use a method overload instead, it should be a
>> pretty
>> >> simple fix.  Luckily, this issue didn't make it into any Geode
>> releases.
>> >>
>> >> Thanks,
>> >> Ryan
>> >>
>> >>
>> >> On Tue, May 14, 2019 at 10:47 AM Udo Kohlmeyer  wrote:
>> >>
>> >>> How did this slip the review process that the signature of a public
>> API
>> >>> was changed?
>> >>>
>> >>> Well done Gester for finding this!!
>> >>>
>> >>> --Udo
>> >>>
>> >>> On 5/14/19 10:40, Dan Smith wrote:
>>  I think the changes for GEODE-6327 broke backwards compatibility in
>>  JSONFormatter with the change from fromJSON(String jsonString) to
>>  fromJSON(String jsonString, String... identityFields)
>> 
>>  Adding an additional varargs parameter to the method breaks code that
>> >> was
>>  compiled against the non-varargs version. I think we need to overload
>> >> the
>>  method instead.
>> 
>>  Thanks to Gester for discovering this with his test!
>> 
>>  -Dan
>> 
>> >>>
>> >>
>>
>>

Re: Backwards compatibility issue with JSONFormatter

[Jared Stewart]

It looks like the japi-compliance-checker tool to which Anthony linked is
available as a gradle plugin: https://github.com/melix/japicmp-gradle-plugin

On Tue, May 14, 2019 at 5:07 PM Ryan McMahon  wrote:

> Fixed in
>
> https://github.com/apache/geode/commit/cc0b37a504480f554b1884491f44a3cca4113ef5
>
> On Tue, May 14, 2019 at 2:06 PM Ryan McMahon  wrote:
>
> > Thanks Anthony!  Have we considered using a compliance checker in our CI
> > like the one in the first link?
> >
> > Side note - I think that varargs is an interesting case that I don't see
> > called out in those links.  Since varargs is just syntactic sugar, it is
> > ultimately the varargs parameter is converted to an array in the
> bytecode.
> > So in the JSON formatter change:
> >
> > fromJSON(byte[] jsonByteArray, String... identityFields)
> >
> >
> > becomes
> >
> > fromJSON(byte[] jsonByteArray, String[] identityFields)
> >
> >
> > Hence the breakage of the ABI.
> >
> > Ryan
> >
> > On Tue, May 14, 2019 at 1:33 PM Anthony Baker  wrote:
> >
> >> Here are a few links on API compatibility:
> >>
> >> https://lvc.github.io/japi-compliance-checker/#Examples
> >> https://wiki.eclipse.org/Evolving_Java-based_APIs
> >> https://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html
> >>
> >>
> >> Anthony
> >>
> >>
> >> > On May 14, 2019, at 12:45 PM, Dan Smith  wrote:
> >> >
> >> > Sounds good! Yeah, this is a bit of an interesting case where code
> will
> >> > still compile against the new API without change, but I think
> >> maintaining
> >> > compatibility of the binary API is also important. Thanks for looking
> >> into
> >> > it.
> >> >
> >> > -Dan
> >> >
> >> > On Tue, May 14, 2019 at 11:22 AM Ryan McMahon 
> >> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> This is my mistake - it was a misunderstanding of what constitutes a
> >> >> breaking public API change.  If a client app were to recompile using
> >> the
> >> >> new bits with the new method signature, there wouldn't be an issue
> >> because
> >> >> the new signature would work with 0 varargs.  The problem arises when
> >> you
> >> >> hot swap the geode dependencies for that client app without
> >> recompiling, as
> >> >> the bytecode no longer matches.  Since we do support the hot swap use
> >> case
> >> >> for CVEs etc, I see now this is considered a breaking API change.
> >> >>
> >> >> I'll change this to use a method overload instead, it should be a
> >> pretty
> >> >> simple fix.  Luckily, this issue didn't make it into any Geode
> >> releases.
> >> >>
> >> >> Thanks,
> >> >> Ryan
> >> >>
> >> >>
> >> >> On Tue, May 14, 2019 at 10:47 AM Udo Kohlmeyer 
> wrote:
> >> >>
> >> >>> How did this slip the review process that the signature of a public
> >> API
> >> >>> was changed?
> >> >>>
> >> >>> Well done Gester for finding this!!
> >> >>>
> >> >>> --Udo
> >> >>>
> >> >>> On 5/14/19 10:40, Dan Smith wrote:
> >>  I think the changes for GEODE-6327 broke backwards compatibility in
> >>  JSONFormatter with the change from fromJSON(String jsonString) to
> >>  fromJSON(String jsonString, String... identityFields)
> >> 
> >>  Adding an additional varargs parameter to the method breaks code
> that
> >> >> was
> >>  compiled against the non-varargs version. I think we need to
> overload
> >> >> the
> >>  method instead.
> >> 
> >>  Thanks to Gester for discovering this with his test!
> >> 
> >>  -Dan
> >> 
> >> >>>
> >> >>
> >>
> >>
>