RE: WAN replication issue in cloud native environments

[Alberto Bustamante Reyes]

I have tried setting the server affinity on the gateway sender's pool in 
AbstractGatewaySender class, when the server location is set, but I dont see 
any difference on the behavior of the connections.

I did not mention that the connections are reset every 5 seconds due to 
"java.io.EOFException: The connection has been reset while reading the header". 
But I dont know yet what is causing it.


De: Dan Smith 
Enviado: martes, 3 de marzo de 2020 18:07
Para: dev@geode.apache.org 
Cc: Bruce Schuchardt ; Jacob Barrett 
; Anilkumar Gingade ; Charlie Black 

Asunto: Re: WAN replication issue in cloud native environments

> We are currently working on other issue related to this change: gw
senders pings are not reaching the gw receivers, so ClientHealthMonitor
closes the connections. I saw that the ping tasks are created by
ServerLocation, so I have tried to solve the issue by changing it to be
done by Endpoint. This change is not finished yet, as in its current status
it causes the closing of connections from gw servers to gw receivers every
5 seconds.

Are you using session affinity? I think you probably will need to since
pings can go over different connections than the data connection.

-Dan

On Tue, Mar 3, 2020 at 3:44 AM Alberto Bustamante Reyes
 wrote:

> Hi Bruce,
>
> Thanks for your comments, but we are not planning to use TLS, so Im afraid
> the PR you are working on will not solve this problem.
>
> The origin of this issue is that we would like to be able to configure all
> gw receivers with the same "hostname-for-senders" value. The reason is that
> we will run a multisite Geode cluster, having each site on a different
> cloud environment, so using just one hostname makes configuration much more
> easier.
>
> When we tried to configure the cluster in this way, we experienced an
> issue with the replication. Using the same hostname-for-senders parameter
> causes that different servers have equals ServerLocation objects, so if one
> receiver is down, the others are considered down too. With the change
> suggested by Jacob this problem is solved, and replication works fine.
>
> We are currently working on other issue related to this change: gw senders
> pings are not reaching the gw receivers, so ClientHealthMonitor closes the
> connections. I saw that the ping tasks are created by ServerLocation, so I
> have tried to solve the issue by changing it to be done by Endpoint. This
> change is not finished yet, as in its current status it causes the closing
> of connections from gw servers to gw receivers every 5 seconds.
>
> Why you dont like the idea of using the InternalDistributedMember for
> distinguish server locations? Are you thinking about other alternative? In
> this use case, two different gw receivers will have the same
> ServerLocation, so we need to distinguish them.
>
> BR/
>
> Alberto B.
>
> 
> De: Bruce Schuchardt 
> Enviado: lunes, 2 de marzo de 2020 20:20
> Para: dev@geode.apache.org ; Jacob Barrett <
> jbarr...@pivotal.io>
> Cc: Anilkumar Gingade ; Charlie Black <
> cbl...@pivotal.io>
> Asunto: Re: WAN replication issue in cloud native environments
>
> I'm coming to this conversation late and probably am missing a lot of
> context.  Is the point of this to be to direct senders to some common
> gateway that all of the gateway receivers are configured to advertise?
> I've been working on a PR to support redirection of connections for
> client/server and gateway communications to a common address and put the
> destination host name in the SNIHostName TLS parameter.  Then you won't
> have to tell servers about the common host name - just tell clients what
> the gateway is and they'll connect to it & tell it what the target host
> name is via the SNIHostName.  However, that only works if SSL is enabled.
>
> PR 4743 is a step toward this approach and changes TcpClient and
> SocketCreator to take an unresolved host address.  After this is merged
> another change will allow folks to set a gateway host/port that will be
> used to form connections and insert the destination hostname into the
> SNIHostName SSLParameter.
>
> I would really like us to avoid including InternalDistributedMembers in
> equality checks for server-locations.  To-date we've only held these
> identifiers in Endpoints and other places for debugging purposes and have
> used ServerLocation to identify servers.
>
> On 1/27/20, 8:56 AM, "Alberto Bustamante Reyes"
>  wrote:
>
> Hi again,
>
> Status update: the simplification of the maps suggested by Jacob made
> useless the new proposed class containing the ServerLocation and the member
> id. With this refactoring, replication is working in the scenario we have
> been discussing in this conversation. Thats great, and I think the code can
> be merged into develop if there are no extra comments in the PR.
>
> But this does not mean we can say that Geode is able to work properly
> when using gw receivers wit

Re: Let's Deprecate the SECURITY_UDP_DHALGO Configuration Property

[Kirk Lund]

+1 to deprecating and removing SECURITY_UDP_DHALGO. The current
implementation is a non-thread-safe hack that would require extensive work
to "fix". It fails EVERY time we run mass test run. I'd prefer to see us
embrace DTLS as a standard.

On Fri, Feb 28, 2020 at 11:43 AM Bill Burcham 
wrote:

> I propose we deprecate Geode’s proprietary UDP message privacy algorithm
> based on the Diffie-Hellman key exchange protocol. This would deprecate:
>
> ConfigurationProperties.SECURITY_UDP_DHALGO
>
> String DistributionConfig.getSecurityUDPDHAlgo()
>
> void DistributionConfig.setSecurityUDPDHAlgo(String attValue)
> DistributionConfig.SECURITY_UDP_DHALGO_NAME
>
> Additionally we’d have to upate documentation to reflect deprecation.
>
> From ConfigurationProperties.java:
>
>
> Application can set this property to valid symmetric key algorithm, to
> encrypt udp messages in Geode. Geode will generate symmetric key using
> Diffie-Hellman key exchange algorithm between peers. That key further used
> by specified algorithm to encrypt the udp messages.
>
> The property (and the feature) was added mid-2016. Unfortunately it was not
> added as an “experimental” feature, so it cannot simply be removed.
>
> Incidentally, the corresponding property for client-server communication,
> SECURITY_CLIENT_DHALGO, is already deprecated. It was deprecated in Geode
> 1.5 in favor of SSL/TLS.
>
> I am proposing deprecating the feature because:
>
>
>1.
>
>The feature has not proven popular with users.
>2.
>
>At least one hard-to-reproduce bug exists in the implementation:
>GEODE-6448 . We’ve
>burned a person-week trying to fix the problem (Bruce Schuchardt and me)
>and it’s not clear how much more time it will take. If we decide to
>deprecate the feature, fixing this problem would be de-prioritized
>accordingly.
>3.
>
>If we decide, in the future, that UDP message security is required, it
>would be better to implement a standard algorithm such as DTLS
>:
>1.
>
>   Our algorithm provides only message privacy whereas DTLS provides
>   privacy, tamper-resistance, and message forgery protection
>   2.
>
>   DTLS is a standard
>   3.
>
>   There is some support for DTLS in the JDK (JEP-219
>    delivered in JDK 9). It’s not a
>   complete implementation e.g. guaranteed delivery is a do-it-yourself
> kit.
>
>
> Actually implementing DTLS is out of scope for this proposal. Adding DTLS
> would be a significant undertaking.
>
> So, how do you feel about me making a GEODE ticket to deprecate the
> SECURITY_UDP_DHALGO configuration property?
>

Help needed to get my PR to pass the stressNewTest

[Eric Shu]

My PR (https://github.com/apache/geode/pull/4709) continue to fail in
stressNewTest. I have been retrigger the test and all failed with same
issue. I need help to find out how to get this PR to pass the target.

Thanks!
-Eric

> Task :geode-assembly:repeatDistributedTest

15:59:04

Do not allow more than 24 test workers

16:50:44

Cannot abort process 'Gradle Test Executor 110' because it is not in
started or detached state

16:50:44

java.lang.IllegalStateException: Cannot abort process 'Gradle Test
Executor 110' because it is not in started or detached state

16:50:44

at 
com.pedjak.gradle.plugins.dockerizedtest.DockerizedExecHandle.abort(DockerizedExecHandle.java:329)

16:50:44

at org.gradle.process.internal.ExecHandle$abort$53.call(Unknown Source)

16:50:44

at 
com.pedjak.gradle.plugins.dockerizedtest.ExitCodeTolerantExecHandle.abort(ExitCodeTolerantExecHandle.groovy:32)

16:50:44

at 
org.gradle.process.internal.worker.DefaultWorkerProcess$2.stop(DefaultWorkerProcess.java:224)

16:50:44

at 
org.gradle.internal.concurrent.CompositeStoppable.stop(CompositeStoppable.java:103)

16:50:44

at 
org.gradle.process.internal.worker.DefaultWorkerProcess.cleanup(DefaultWorkerProcess.java:232)

16:50:44

at 
org.gradle.process.internal.worker.DefaultWorkerProcess.start(DefaultWorkerProcess.java:166)

16:50:44

at 
org.gradle.process.internal.worker.DefaultWorkerProcessBuilder$MemoryRequestingWorkerProcess.start(DefaultWorkerProcessBuilder.java:221)

16:50:44

at 
org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.forkProcess(ForkingTestClassProcessor.java:108)

16:50:44

at 
org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.processTestClass(ForkingTestClassProcessor.java:84)

16:50:44

at 
org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.processTestClass(RestartEveryNTestClassProcessor.java:52)

16:50:44

at sun.reflect.GeneratedMethodAccessor569.invoke(Unknown Source)

16:50:44

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

16:50:44

at java.lang.reflect.Method.invoke(Method.java:498)

16:50:44

at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)

16:50:44

at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)

16:50:44

at 
org.gradle.internal.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:29)

16:50:44

at 
org.gradle.internal.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:87)

16:50:44

at 
org.gradle.internal.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:36)

16:50:44

at org.gradle.internal.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:71)

16:50:44

at 
org.gradle.internal.concurrent.InterruptibleRunnable.run(InterruptibleRunnable.java:42)

16:50:44

at 
org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)

16:50:44

at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)

16:50:44

at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)

16:50:44

at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

16:50:44

at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

16:50:44

at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)

16:50:44

at java.lang.Thread.run(Thread.java:748)

Re: Help needed to get my PR to pass the stressNewTest

[Mark Hanson]

Hi Eric, 

When you have a moment, ping me. I can help. I think…

Thanks,
Mark

> On Mar 4, 2020, at 1:27 PM, Eric Shu  wrote:
> 
> My PR (https://github.com/apache/geode/pull/4709) continue to fail in
> stressNewTest. I have been retrigger the test and all failed with same
> issue. I need help to find out how to get this PR to pass the target.
> 
> Thanks!
> -Eric
> 
>> Task :geode-assembly:repeatDistributedTest
> 
> 15:59:04
> 
> Do not allow more than 24 test workers
> 
> 16:50:44
> 
> Cannot abort process 'Gradle Test Executor 110' because it is not in
> started or detached state
> 
> 16:50:44
> 
> java.lang.IllegalStateException: Cannot abort process 'Gradle Test
> Executor 110' because it is not in started or detached state
> 
> 16:50:44
> 
>at 
> com.pedjak.gradle.plugins.dockerizedtest.DockerizedExecHandle.abort(DockerizedExecHandle.java:329)
> 
> 16:50:44
> 
>at org.gradle.process.internal.ExecHandle$abort$53.call(Unknown Source)
> 
> 16:50:44
> 
>at 
> com.pedjak.gradle.plugins.dockerizedtest.ExitCodeTolerantExecHandle.abort(ExitCodeTolerantExecHandle.groovy:32)
> 
> 16:50:44
> 
>at 
> org.gradle.process.internal.worker.DefaultWorkerProcess$2.stop(DefaultWorkerProcess.java:224)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.concurrent.CompositeStoppable.stop(CompositeStoppable.java:103)
> 
> 16:50:44
> 
>at 
> org.gradle.process.internal.worker.DefaultWorkerProcess.cleanup(DefaultWorkerProcess.java:232)
> 
> 16:50:44
> 
>at 
> org.gradle.process.internal.worker.DefaultWorkerProcess.start(DefaultWorkerProcess.java:166)
> 
> 16:50:44
> 
>at 
> org.gradle.process.internal.worker.DefaultWorkerProcessBuilder$MemoryRequestingWorkerProcess.start(DefaultWorkerProcessBuilder.java:221)
> 
> 16:50:44
> 
>at 
> org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.forkProcess(ForkingTestClassProcessor.java:108)
> 
> 16:50:44
> 
>at 
> org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.processTestClass(ForkingTestClassProcessor.java:84)
> 
> 16:50:44
> 
>at 
> org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.processTestClass(RestartEveryNTestClassProcessor.java:52)
> 
> 16:50:44
> 
>at sun.reflect.GeneratedMethodAccessor569.invoke(Unknown Source)
> 
> 16:50:44
> 
>at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 
> 16:50:44
> 
>at java.lang.reflect.Method.invoke(Method.java:498)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:29)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:87)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:36)
> 
> 16:50:44
> 
>at org.gradle.internal.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:71)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.concurrent.InterruptibleRunnable.run(InterruptibleRunnable.java:42)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> 
> 16:50:44
> 
>at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 
> 16:50:44
> 
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 
> 16:50:44
> 
>at 
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> 
> 16:50:44
> 
>at java.lang.Thread.run(Thread.java:748)

Re: Help needed to get my PR to pass the stressNewTest

[Jinmei Liao]

Have you tried running all the changed tests in your IDEA repeatedly to see
if there would be a failure?

On Wed, Mar 4, 2020 at 1:28 PM Eric Shu  wrote:

> My PR (https://github.com/apache/geode/pull/4709) continue to fail in
> stressNewTest. I have been retrigger the test and all failed with same
> issue. I need help to find out how to get this PR to pass the target.
>
> Thanks!
> -Eric
>
> > Task :geode-assembly:repeatDistributedTest
>
> 15:59:04
>
> Do not allow more than 24 test workers
>
> 16:50:44
>
> Cannot abort process 'Gradle Test Executor 110' because it is not in
> started or detached state
>
> 16:50:44
>
> java.lang.IllegalStateException: Cannot abort process 'Gradle Test
> Executor 110' because it is not in started or detached state
>
> 16:50:44
>
> at
> com.pedjak.gradle.plugins.dockerizedtest.DockerizedExecHandle.abort(DockerizedExecHandle.java:329)
>
> 16:50:44
>
> at org.gradle.process.internal.ExecHandle$abort$53.call(Unknown Source)
>
> 16:50:44
>
> at
> com.pedjak.gradle.plugins.dockerizedtest.ExitCodeTolerantExecHandle.abort(ExitCodeTolerantExecHandle.groovy:32)
>
> 16:50:44
>
> at
> org.gradle.process.internal.worker.DefaultWorkerProcess$2.stop(DefaultWorkerProcess.java:224)
>
> 16:50:44
>
> at
> org.gradle.internal.concurrent.CompositeStoppable.stop(CompositeStoppable.java:103)
>
> 16:50:44
>
> at
> org.gradle.process.internal.worker.DefaultWorkerProcess.cleanup(DefaultWorkerProcess.java:232)
>
> 16:50:44
>
> at
> org.gradle.process.internal.worker.DefaultWorkerProcess.start(DefaultWorkerProcess.java:166)
>
> 16:50:44
>
> at
> org.gradle.process.internal.worker.DefaultWorkerProcessBuilder$MemoryRequestingWorkerProcess.start(DefaultWorkerProcessBuilder.java:221)
>
> 16:50:44
>
> at
> org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.forkProcess(ForkingTestClassProcessor.java:108)
>
> 16:50:44
>
> at
> org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.processTestClass(ForkingTestClassProcessor.java:84)
>
> 16:50:44
>
> at
> org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.processTestClass(RestartEveryNTestClassProcessor.java:52)
>
> 16:50:44
>
> at sun.reflect.GeneratedMethodAccessor569.invoke(Unknown Source)
>
> 16:50:44
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> 16:50:44
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> 16:50:44
>
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>
> 16:50:44
>
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>
> 16:50:44
>
> at
> org.gradle.internal.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:29)
>
> 16:50:44
>
> at
> org.gradle.internal.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:87)
>
> 16:50:44
>
> at
> org.gradle.internal.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:36)
>
> 16:50:44
>
> at
> org.gradle.internal.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:71)
>
> 16:50:44
>
> at
> org.gradle.internal.concurrent.InterruptibleRunnable.run(InterruptibleRunnable.java:42)
>
> 16:50:44
>
> at
> org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
>
> 16:50:44
>
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
>
> 16:50:44
>
> at
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
>
> 16:50:44
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> 16:50:44
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> 16:50:44
>
> at
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
>
> 16:50:44
>
> at java.lang.Thread.run(Thread.java:748)
>


-- 
Cheers

Jinmei

Re: Help needed to get my PR to pass the stressNewTest

[Eric Shu]

Running repeated tests in IDEA takes a while. So no, I have not finished
running of the test. Will try again to see if failed with the same.

-Eric

On Wed, Mar 4, 2020 at 2:53 PM Jinmei Liao  wrote:

> Have you tried running all the changed tests in your IDEA repeatedly to see
> if there would be a failure?
>
> On Wed, Mar 4, 2020 at 1:28 PM Eric Shu  wrote:
>
> > My PR (https://github.com/apache/geode/pull/4709) continue to fail in
> > stressNewTest. I have been retrigger the test and all failed with same
> > issue. I need help to find out how to get this PR to pass the target.
> >
> > Thanks!
> > -Eric
> >
> > > Task :geode-assembly:repeatDistributedTest
> >
> > 15:59:04
> >
> > Do not allow more than 24 test workers
> >
> > 16:50:44
> >
> > Cannot abort process 'Gradle Test Executor 110' because it is not in
> > started or detached state
> >
> > 16:50:44
> >
> > java.lang.IllegalStateException: Cannot abort process 'Gradle Test
> > Executor 110' because it is not in started or detached state
> >
> > 16:50:44
> >
> > at
> >
> com.pedjak.gradle.plugins.dockerizedtest.DockerizedExecHandle.abort(DockerizedExecHandle.java:329)
> >
> > 16:50:44
> >
> > at org.gradle.process.internal.ExecHandle$abort$53.call(Unknown
> Source)
> >
> > 16:50:44
> >
> > at
> >
> com.pedjak.gradle.plugins.dockerizedtest.ExitCodeTolerantExecHandle.abort(ExitCodeTolerantExecHandle.groovy:32)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.process.internal.worker.DefaultWorkerProcess$2.stop(DefaultWorkerProcess.java:224)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.concurrent.CompositeStoppable.stop(CompositeStoppable.java:103)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.process.internal.worker.DefaultWorkerProcess.cleanup(DefaultWorkerProcess.java:232)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.process.internal.worker.DefaultWorkerProcess.start(DefaultWorkerProcess.java:166)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.process.internal.worker.DefaultWorkerProcessBuilder$MemoryRequestingWorkerProcess.start(DefaultWorkerProcessBuilder.java:221)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.forkProcess(ForkingTestClassProcessor.java:108)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.processTestClass(ForkingTestClassProcessor.java:84)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.processTestClass(RestartEveryNTestClassProcessor.java:52)
> >
> > 16:50:44
> >
> > at sun.reflect.GeneratedMethodAccessor569.invoke(Unknown Source)
> >
> > 16:50:44
> >
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > 16:50:44
> >
> > at java.lang.reflect.Method.invoke(Method.java:498)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:29)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:87)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:36)
> >
> > 16:50:44
> >
> > at
> > org.gradle.internal.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:71)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.concurrent.InterruptibleRunnable.run(InterruptibleRunnable.java:42)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
> >
> > 16:50:44
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> > 16:50:44
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >
> > 16:50:44
> >
> > at
> >
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
> >
> > 16:50:44
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
>
>
> --
> Cheers
>
> Jinmei
>

Cloudburst from RISELabs

[Sai Boorlagadda]

Devs,

I came across Cloudburst paper which introduces function execution
integrated into a KV store (Anna) by Berkley folks @ RISELabs which
resembles Geode's functions execution engine.

I thought I would share this paper with the rest of the community.

https://arxiv.org/pdf/2001.04592.pdf

Sai

Re: WAN replication issue in cloud native environments

[Dan Smith]

Oh, sorry, I meant server affinity with the proxy itself. So that it will
always route traffic from the same gateway sender to the same gateway
receiver. Hopefully that would ensure that pings go to the same receiver
data is sent to.

-Dan

On Wed, Mar 4, 2020, 1:31 AM Alberto Bustamante Reyes
 wrote:

> I have tried setting the server affinity on the gateway sender's pool in
> AbstractGatewaySender class, when the server location is set, but I dont
> see any difference on the behavior of the connections.
>
> I did not mention that the connections are reset every 5 seconds due to
> "java.io.EOFException: The connection has been reset while reading the
> header". But I dont know yet what is causing it.
>
> --
> *De:* Dan Smith 
> *Enviado:* martes, 3 de marzo de 2020 18:07
> *Para:* dev@geode.apache.org 
> *Cc:* Bruce Schuchardt ; Jacob Barrett <
> jbarr...@pivotal.io>; Anilkumar Gingade ; Charlie
> Black 
> *Asunto:* Re: WAN replication issue in cloud native environments
>
> > We are currently working on other issue related to this change: gw
> senders pings are not reaching the gw receivers, so ClientHealthMonitor
> closes the connections. I saw that the ping tasks are created by
> ServerLocation, so I have tried to solve the issue by changing it to be
> done by Endpoint. This change is not finished yet, as in its current status
> it causes the closing of connections from gw servers to gw receivers every
> 5 seconds.
>
> Are you using session affinity? I think you probably will need to since
> pings can go over different connections than the data connection.
>
> -Dan
>
> On Tue, Mar 3, 2020 at 3:44 AM Alberto Bustamante Reyes
>  wrote:
>
> > Hi Bruce,
> >
> > Thanks for your comments, but we are not planning to use TLS, so Im
> afraid
> > the PR you are working on will not solve this problem.
> >
> > The origin of this issue is that we would like to be able to configure
> all
> > gw receivers with the same "hostname-for-senders" value. The reason is
> that
> > we will run a multisite Geode cluster, having each site on a different
> > cloud environment, so using just one hostname makes configuration much
> more
> > easier.
> >
> > When we tried to configure the cluster in this way, we experienced an
> > issue with the replication. Using the same hostname-for-senders parameter
> > causes that different servers have equals ServerLocation objects, so if
> one
> > receiver is down, the others are considered down too. With the change
> > suggested by Jacob this problem is solved, and replication works fine.
> >
> > We are currently working on other issue related to this change: gw
> senders
> > pings are not reaching the gw receivers, so ClientHealthMonitor closes
> the
> > connections. I saw that the ping tasks are created by ServerLocation, so
> I
> > have tried to solve the issue by changing it to be done by Endpoint. This
> > change is not finished yet, as in its current status it causes the
> closing
> > of connections from gw servers to gw receivers every 5 seconds.
> >
> > Why you dont like the idea of using the InternalDistributedMember for
> > distinguish server locations? Are you thinking about other alternative?
> In
> > this use case, two different gw receivers will have the same
> > ServerLocation, so we need to distinguish them.
> >
> > BR/
> >
> > Alberto B.
> >
> > 
> > De: Bruce Schuchardt 
> > Enviado: lunes, 2 de marzo de 2020 20:20
> > Para: dev@geode.apache.org ; Jacob Barrett <
> > jbarr...@pivotal.io>
> > Cc: Anilkumar Gingade ; Charlie Black <
> > cbl...@pivotal.io>
> > Asunto: Re: WAN replication issue in cloud native environments
> >
> > I'm coming to this conversation late and probably am missing a lot of
> > context.  Is the point of this to be to direct senders to some common
> > gateway that all of the gateway receivers are configured to advertise?
> > I've been working on a PR to support redirection of connections for
> > client/server and gateway communications to a common address and put the
> > destination host name in the SNIHostName TLS parameter.  Then you won't
> > have to tell servers about the common host name - just tell clients what
> > the gateway is and they'll connect to it & tell it what the target host
> > name is via the SNIHostName.  However, that only works if SSL is enabled.
> >
> > PR 4743 is a step toward this approach and changes TcpClient and
> > SocketCreator to take an unresolved host address.  After this is merged
> > another change will allow folks to set a gateway host/port that will be
> > used to form connections and insert the destination hostname into the
> > SNIHostName SSLParameter.
> >
> > I would really like us to avoid including InternalDistributedMembers in
> > equality checks for server-locations.  To-date we've only held these
> > identifiers in Endpoints and other places for debugging purposes and have
> > used ServerLocation to identify servers.
> >
> > On 1/27/20, 8:56 AM, "Alb