Odg: gateway sender queue
Hi, thanks for answers. Some more details regarding 1st question. Is this behavior same (for serial and parallel gateway sender) in case queue is persistent? Meaning, should queue (persistent) be purged if we restart gateway sender? Thanks, Mario Šalje: Dan Smith Poslano: 5. studenog 2019. 18:52 Prima: dev@geode.apache.org Predmet: Re: gateway sender queue Some replies, inline: * During testing we have observed, different behavior in parallel and > serial gateway senders. In case we manually stop, than start gateway > senders, for parallel gateway senders, queue is purged, but for serial > gateway senders this is not the case. Is this normal behavior or bug? > Hmm, I also think stop is supposed to clear the queue. I think if you are seeing that it doesn't clear the queue, that might be a bug. > * What happens with the queues when whole cluster is stopped and later > started (In our tests with persistent queues, the events are kept)? > Persistent queues will keep all of the events when you restart. > * Could we add extra option in gfsh command "start gateway sender" > that allows to control queues reset (for instance --cleanQueues)? > If stop does clear the queue, would this be needed? It might still be reasonable - I've heard folks request a way to clear running queues as well. -Dan
Update geode-native-build image as part of release process
Hi all, Some time ago I opened GEODE-7056 to update the Dockerfile of the "geode-native-build", because I saw that it was using 1.6.0 although 1.9.0 was available. After this change, other ticket was needed to build and update the image in Dockerhub. I think this task (update file, build image and update image) should be part of the release process. Otherwise, the image will be outdated once there is a new release. And now that 1.11.0 is closer, I think its a good moment to do it. What do you think? BR/ Alberto B.
geode-native integration tests
Hi Geode community, Im curious about the geode-native integration tests. I have seen there are two kind of tests, depending on the framework they are based on. But no one of them are included in the CI executed by travis. Are they executed only as part of the Geode release process? What is the way of working regarding new tests? I assume that new tests should be written using the new framework. But what about the old ones? Is there any plan to port these test to the new framework? (I suppose that PRs for that are welcome) What if a PR requires the modification of an old integration test? I suppose that as a general rule, instead of modifying that test case, a new test case in the new framework should be written and then the old one should be removed. Im missing all this info about the way of working in the CONTRIBUTING.md file, I can include it once this is cleared up. Thanks! Alberto B.
Re: Update geode-native-build image as part of release process
+1 On Thu, Nov 7, 2019 at 6:46 AM Alberto Bustamante Reyes wrote: > Hi all, > > Some time ago I opened GEODE-7056 to update the Dockerfile of the > "geode-native-build", because I saw that it was using 1.6.0 although 1.9.0 > was available. After this change, other ticket was needed to build and > update the image in Dockerhub. > > I think this task (update file, build image and update image) should be > part of the release process. Otherwise, the image will be outdated once > there is a new release. And now that 1.11.0 is closer, I think its a good > moment to do it. > > What do you think? > > BR/ > > Alberto B. >
Re: geode-native integration tests
> On Nov 7, 2019, at 7:41 AM, Alberto Bustamante Reyes > wrote: > > Im curious about the geode-native integration tests. I have seen there are > two kind of tests, depending on the framework they are based on. But no one > of them are included in the CI executed by travis. Are they executed only as > part of the Geode release process? The long term goal is to have geode-native built continuously with along with main java bits. If you are interested in helping with this effort that might speed up the process. > What is the way of working regarding new tests? I assume that new tests > should be written using the new framework. But what about the old ones? Is > there any plan to port these test to the new framework? (I suppose that PRs > for that are welcome) Yes all new tests should be written in the new framework. The new framework is a work in progress itself. It it is missing something, add it. If it needs refactoring, refactor it. There is no current plan to bulk convert the old tests, see next response. > What if a PR requires the modification of an old integration test? I suppose > that as a general rule, instead of modifying that test case, a new test case > in the new framework should be written and then the old one should be removed. The old framework should be considered deprecated and obsolete. Make minimal changes as necessary to it. If you make a PR that touches several tests it is reasonable to just update the old integration tests. If your PR would have you changing one or a few of the old tests then please rewrite those tests in the new framework and delete the old. Hopefully over time we won’t have any old tests left. > Im missing all this info about the way of working in the CONTRIBUTING.md > file, I can include it once this is cleared up. Yes please contribute to the CONTRIBUTING.md! Thanks, Jake
Re: Update geode-native-build image as part of release process
+1 On Thu, Nov 7, 2019 at 8:38 AM Owen Nichols wrote: > +1 > > On Thu, Nov 7, 2019 at 6:46 AM Alberto Bustamante Reyes > wrote: > > > Hi all, > > > > Some time ago I opened GEODE-7056 to update the Dockerfile of the > > "geode-native-build", because I saw that it was using 1.6.0 although > 1.9.0 > > was available. After this change, other ticket was needed to build and > > update the image in Dockerhub. > > > > I think this task (update file, build image and update image) should be > > part of the release process. Otherwise, the image will be outdated once > > there is a new release. And now that 1.11.0 is closer, I think its a good > > moment to do it. > > > > What do you think? > > > > BR/ > > > > Alberto B. > > >
Re: bug fix needed for release/1.11.0
The change passed in SSL benchmark testing and can be cherry-picked into the release branch. The Benchmark job as a whole failed due to perf degradation with the Security Manager, but that's a different set of tests. On 11/6/19 3:57 PM, Helena Bales wrote: +1 to cherry-picking the fix. The sha hasn't made it to benchmarks yet due to an issue with CI losing resource refs that were needed to keep it moving through the pipeline. The next commit is still about an hour away from triggering benchmarks. In my manual benchmarking of this change, I found that it resolved the issue with SSL and passed the benchmarks. Obviously we still need to confirm that it works through the main pipeline, but I feel confident that it will pass the benchmark job. Thanks, Helena Bales (they/them) On Wed, Nov 6, 2019 at 9:28 AM Mark Hanson wrote: Any other votes? I have 2 people in favor. Voting will close at noon. Thanks, Mark On Nov 6, 2019, at 8:00 AM, Bruce Schuchardt wrote: The fix for this problem is in the CI pipeline today: https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Build/builds/1341 On 11/5/19 10:49 AM, Owen Nichols wrote: +1 for bringing this fix to release/1.11.0 (after it has passed Benchmarks on develop) On Nov 5, 2019, at 10:45 AM, Bruce Schuchardt wrote: The PR for GEODE-6661 introduced a problem in SSL communications that needs to be fixed. It changed SSL handshakes to use a temporary buffer that's discarded when the handshake completes, but sometimes this buffer contains application data that must be retained. This seems to be causing our Benchmark SSL test failures in CI. I'm preparing a fix. We can either revert the PR for GEODE-6661 on that branch or cherry-pick the correction when it's ready.
Re: bug fix needed for release/1.11.0
+1 to include the fix On Thu, Nov 7, 2019 at 8:54 AM Bruce Schuchardt wrote: > The change passed in SSL benchmark testing and can be cherry-picked into > the release branch. The Benchmark job as a whole failed due to perf > degradation with the Security Manager, but that's a different set of tests. > > > On 11/6/19 3:57 PM, Helena Bales wrote: > > +1 to cherry-picking the fix. > > > > The sha hasn't made it to benchmarks yet due to an issue with CI losing > > resource refs that were needed to keep it moving through the pipeline. > The > > next commit is still about an hour away from triggering benchmarks. > > In my manual benchmarking of this change, I found that it resolved the > > issue with SSL and passed the benchmarks. Obviously we still need to > > confirm that it works through the main pipeline, but I feel confident > that > > it will pass the benchmark job. > > > > Thanks, > > Helena Bales (they/them) > > > > On Wed, Nov 6, 2019 at 9:28 AM Mark Hanson wrote: > > > >> Any other votes? I have 2 people in favor. > >> > >> Voting will close at noon. > >> > >> Thanks, > >> Mark > >> > >>> On Nov 6, 2019, at 8:00 AM, Bruce Schuchardt > >> wrote: > >>> The fix for this problem is in the CI pipeline today: > >> > https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Build/builds/1341 > >>> On 11/5/19 10:49 AM, Owen Nichols wrote: > +1 for bringing this fix to release/1.11.0 (after it has passed > >> Benchmarks on develop) > > On Nov 5, 2019, at 10:45 AM, Bruce Schuchardt < > bschucha...@pivotal.io> > >> wrote: > > The PR for GEODE-6661 introduced a problem in SSL communications that > >> needs to be fixed. It changed SSL handshakes to use a temporary buffer > >> that's discarded when the handshake completes, but sometimes this buffer > >> contains application data that must be retained. This seems to be > causing > >> our Benchmark SSL test failures in CI. > > I'm preparing a fix. We can either revert the PR for GEODE-6661 on > >> that branch or cherry-pick the correction when it's ready. > >> >
Re: bug fix needed for release/1.11.0
Based on the lack of “-1” votes, I have cherry-picked the change… Thanks, Mark > On Nov 7, 2019, at 8:54 AM, Bruce Schuchardt wrote: > > The change passed in SSL benchmark testing and can be cherry-picked into the > release branch. The Benchmark job as a whole failed due to perf degradation > with the Security Manager, but that's a different set of tests. > > > On 11/6/19 3:57 PM, Helena Bales wrote: >> +1 to cherry-picking the fix. >> >> The sha hasn't made it to benchmarks yet due to an issue with CI losing >> resource refs that were needed to keep it moving through the pipeline. The >> next commit is still about an hour away from triggering benchmarks. >> In my manual benchmarking of this change, I found that it resolved the >> issue with SSL and passed the benchmarks. Obviously we still need to >> confirm that it works through the main pipeline, but I feel confident that >> it will pass the benchmark job. >> >> Thanks, >> Helena Bales (they/them) >> >> On Wed, Nov 6, 2019 at 9:28 AM Mark Hanson wrote: >> >>> Any other votes? I have 2 people in favor. >>> >>> Voting will close at noon. >>> >>> Thanks, >>> Mark >>> On Nov 6, 2019, at 8:00 AM, Bruce Schuchardt >>> wrote: The fix for this problem is in the CI pipeline today: >>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Build/builds/1341 On 11/5/19 10:49 AM, Owen Nichols wrote: > +1 for bringing this fix to release/1.11.0 (after it has passed >>> Benchmarks on develop) >> On Nov 5, 2019, at 10:45 AM, Bruce Schuchardt >>> wrote: >> The PR for GEODE-6661 introduced a problem in SSL communications that >>> needs to be fixed. It changed SSL handshakes to use a temporary buffer >>> that's discarded when the handshake completes, but sometimes this buffer >>> contains application data that must be retained. This seems to be causing >>> our Benchmark SSL test failures in CI. >> I'm preparing a fix. We can either revert the PR for GEODE-6661 on >>> that branch or cherry-pick the correction when it's ready. >>>
Re: Lucene upgrade
Gester, I don't think we need to write in the old format, we just need the new format not to be written while old members can potentially read the lucene files. Option 1 can be very similar to Dan's snippet of code. I think Option 2 is going to leave a lot of people unhappy when they get stuck with what Mario is experiencing right now and all we can say is "you should have read the doc". Not to say Option 2 isn't valid and it's definitely the least amount of work to do, I still vote option 1. On Wed, Nov 6, 2019 at 5:16 PM Xiaojian Zhou wrote: > Usually re-creating region and index are expensive and customers are > reluctant to do it, according to my memory. > > We do have an offline reindex scripts or steps (written by Barry?). If that > could be an option, they can try that offline tool. > > I saw from Mario's email, he said: "I didn't found a way to write lucene in > older format. They only support > reading old format indexes with newer version by using lucene-backward- > codec." > > That's why I think option-1 is not feasible. > > Option-2 will cause the queue to be filled. But usually customer will hold > on, silence or reduce their business throughput when > doing rolling upgrade. I wonder if it's a reasonable assumption. > > Overall, after compared all the 3 options, I still think option-2 is the > best bet. > > Regards > Gester > > > On Wed, Nov 6, 2019 at 3:38 PM Jacob Barrett wrote: > > > > > > > > On Nov 6, 2019, at 3:36 PM, Jason Huynh wrote: > > > > > > Jake - there is a side effect to this in that the user would have to > > > reimport all their data into the user defined region too. Client apps > > > would also have to know which of the regions to put into.. also, I may > be > > > misunderstanding this suggestion, completely. In either case, I'll > > support > > > whoever implements the changes :-P > > > > Ah… there isn’t a way to re-index the existing data. Eh… just a thought. > > > > -Jake > > > > >
Re: Lucene upgrade
Oh, I misunderstood option-1 and option-2. What I vote is Jason's option-1. On Thu, Nov 7, 2019 at 9:19 AM Jason Huynh wrote: > Gester, I don't think we need to write in the old format, we just need the > new format not to be written while old members can potentially read the > lucene files. Option 1 can be very similar to Dan's snippet of code. > > I think Option 2 is going to leave a lot of people unhappy when they get > stuck with what Mario is experiencing right now and all we can say is "you > should have read the doc". Not to say Option 2 isn't valid and it's > definitely the least amount of work to do, I still vote option 1. > > On Wed, Nov 6, 2019 at 5:16 PM Xiaojian Zhou wrote: > > > Usually re-creating region and index are expensive and customers are > > reluctant to do it, according to my memory. > > > > We do have an offline reindex scripts or steps (written by Barry?). If > that > > could be an option, they can try that offline tool. > > > > I saw from Mario's email, he said: "I didn't found a way to write lucene > in > > older format. They only support > > reading old format indexes with newer version by using lucene-backward- > > codec." > > > > That's why I think option-1 is not feasible. > > > > Option-2 will cause the queue to be filled. But usually customer will > hold > > on, silence or reduce their business throughput when > > doing rolling upgrade. I wonder if it's a reasonable assumption. > > > > Overall, after compared all the 3 options, I still think option-2 is the > > best bet. > > > > Regards > > Gester > > > > > > On Wed, Nov 6, 2019 at 3:38 PM Jacob Barrett > wrote: > > > > > > > > > > > > On Nov 6, 2019, at 3:36 PM, Jason Huynh wrote: > > > > > > > > Jake - there is a side effect to this in that the user would have to > > > > reimport all their data into the user defined region too. Client > apps > > > > would also have to know which of the regions to put into.. also, I > may > > be > > > > misunderstanding this suggestion, completely. In either case, I'll > > > support > > > > whoever implements the changes :-P > > > > > > Ah… there isn’t a way to re-index the existing data. Eh… just a > thought. > > > > > > -Jake > > > > > > > > >
RE: geode-native integration tests
Thanks for the answer Jake, I have sent a PR with the changes. I already know the plans to use concourse instead of travis in the Geode client, I have it in mind, its a good opportunity to learn about concourse. De: Jacob Barrett Enviado: jueves, 7 de noviembre de 2019 17:50 Para: dev@geode.apache.org Asunto: Re: geode-native integration tests > On Nov 7, 2019, at 7:41 AM, Alberto Bustamante Reyes > wrote: > > Im curious about the geode-native integration tests. I have seen there are > two kind of tests, depending on the framework they are based on. But no one > of them are included in the CI executed by travis. Are they executed only as > part of the Geode release process? The long term goal is to have geode-native built continuously with along with main java bits. If you are interested in helping with this effort that might speed up the process. > What is the way of working regarding new tests? I assume that new tests > should be written using the new framework. But what about the old ones? Is > there any plan to port these test to the new framework? (I suppose that PRs > for that are welcome) Yes all new tests should be written in the new framework. The new framework is a work in progress itself. It it is missing something, add it. If it needs refactoring, refactor it. There is no current plan to bulk convert the old tests, see next response. > What if a PR requires the modification of an old integration test? I suppose > that as a general rule, instead of modifying that test case, a new test case > in the new framework should be written and then the old one should be removed. The old framework should be considered deprecated and obsolete. Make minimal changes as necessary to it. If you make a PR that touches several tests it is reasonable to just update the old integration tests. If your PR would have you changing one or a few of the old tests then please rewrite those tests in the new framework and delete the old. Hopefully over time we won’t have any old tests left. > Im missing all this info about the way of working in the CONTRIBUTING.md > file, I can include it once this is cleared up. Yes please contribute to the CONTRIBUTING.md! Thanks, Jake
Re: [DISCUSS] - Upgrading from Spring 4 to Spring 5
+1 On Wed, Oct 30, 2019 at 1:39 PM Udo Kohlmeyer wrote: > Sorry, > > To clarify... When we change the Spring version we would be looking at > looking to use the latest version and it's associated BOM. > > That might be inclusive of other Spring project upgrades. > > --Udo > > On 10/30/19 1:35 PM, Nabarun Nag wrote: > > Hi Udo, > > Maven has the latest as 5.2.0.RELEASE as the latest version. In the > > Dependency.groovy file, we have been putting the full version number. > Hence > > I am guessing you are suggesting we put 5.2.0.RELEASE? > > > > What about the status of the following dependencies? > > > > 'org.springframework.hateoas', name: 'spring-hateoas', version: > > '0.25.0.RELEASE' > > 'org.springframework.ldap', name: 'spring-ldap-core', version: > > '2.3.2.RELEASE' > > 'org.springframework.shell', name: 'spring-shell', version: > '1.2.0.RELEASE' > > > > Regards > > Naba > > >
Re: [DISCUSS] - Upgrading from Spring 4 to Spring 5
+1 On Thu, Nov 7, 2019 at 12:49 PM Jens Deppe wrote: > +1 > > On Wed, Oct 30, 2019 at 1:39 PM Udo Kohlmeyer wrote: > > > Sorry, > > > > To clarify... When we change the Spring version we would be looking at > > looking to use the latest version and it's associated BOM. > > > > That might be inclusive of other Spring project upgrades. > > > > --Udo > > > > On 10/30/19 1:35 PM, Nabarun Nag wrote: > > > Hi Udo, > > > Maven has the latest as 5.2.0.RELEASE as the latest version. In the > > > Dependency.groovy file, we have been putting the full version number. > > Hence > > > I am guessing you are suggesting we put 5.2.0.RELEASE? > > > > > > What about the status of the following dependencies? > > > > > > 'org.springframework.hateoas', name: 'spring-hateoas', version: > > > '0.25.0.RELEASE' > > > 'org.springframework.ldap', name: 'spring-ldap-core', version: > > > '2.3.2.RELEASE' > > > 'org.springframework.shell', name: 'spring-shell', version: > > '1.2.0.RELEASE' > > > > > > Regards > > > Naba > > > > > >
Re: [DISCUSS] - Upgrading from Spring 4 to Spring 5
+1 On Thu, Nov 7, 2019 at 1:28 PM Dan Smith wrote: > +1 > > On Thu, Nov 7, 2019 at 12:49 PM Jens Deppe wrote: > > > +1 > > > > On Wed, Oct 30, 2019 at 1:39 PM Udo Kohlmeyer wrote: > > > > > Sorry, > > > > > > To clarify... When we change the Spring version we would be looking at > > > looking to use the latest version and it's associated BOM. > > > > > > That might be inclusive of other Spring project upgrades. > > > > > > --Udo > > > > > > On 10/30/19 1:35 PM, Nabarun Nag wrote: > > > > Hi Udo, > > > > Maven has the latest as 5.2.0.RELEASE as the latest version. In the > > > > Dependency.groovy file, we have been putting the full version number. > > > Hence > > > > I am guessing you are suggesting we put 5.2.0.RELEASE? > > > > > > > > What about the status of the following dependencies? > > > > > > > > 'org.springframework.hateoas', name: 'spring-hateoas', version: > > > > '0.25.0.RELEASE' > > > > 'org.springframework.ldap', name: 'spring-ldap-core', version: > > > > '2.3.2.RELEASE' > > > > 'org.springframework.shell', name: 'spring-shell', version: > > > '1.2.0.RELEASE' > > > > > > > > Regards > > > > Naba > > > > > > > > > >
Re: [DISCUSS] - Upgrading from Spring 4 to Spring 5
Initial attempts have been made : https://github.com/apache/geode/pull/4256 This PR has Maintainers edit permissions enabled. We need to figure out a plan on these following springframework dependencies too. - spring-hateos [geode - 0.25.0 RELEASE latest 1.0.1 RELEASE] - spring-plugin-core [geode - 1.2.0 RELEASE latest 2.0.0 RELEASE] - spring-plugin-metadata [geode - 1.2.0 RELEASE latest 2.0.0 RELEASE] On Thu, Nov 7, 2019 at 2:43 PM Jason Huynh wrote: > +1 > > On Thu, Nov 7, 2019 at 1:28 PM Dan Smith wrote: > > > +1 > > > > On Thu, Nov 7, 2019 at 12:49 PM Jens Deppe wrote: > > > > > +1 > > > > > > On Wed, Oct 30, 2019 at 1:39 PM Udo Kohlmeyer wrote: > > > > > > > Sorry, > > > > > > > > To clarify... When we change the Spring version we would be looking > at > > > > looking to use the latest version and it's associated BOM. > > > > > > > > That might be inclusive of other Spring project upgrades. > > > > > > > > --Udo > > > > > > > > On 10/30/19 1:35 PM, Nabarun Nag wrote: > > > > > Hi Udo, > > > > > Maven has the latest as 5.2.0.RELEASE as the latest version. In > the > > > > > Dependency.groovy file, we have been putting the full version > number. > > > > Hence > > > > > I am guessing you are suggesting we put 5.2.0.RELEASE? > > > > > > > > > > What about the status of the following dependencies? > > > > > > > > > > 'org.springframework.hateoas', name: 'spring-hateoas', version: > > > > > '0.25.0.RELEASE' > > > > > 'org.springframework.ldap', name: 'spring-ldap-core', version: > > > > > '2.3.2.RELEASE' > > > > > 'org.springframework.shell', name: 'spring-shell', version: > > > > '1.2.0.RELEASE' > > > > > > > > > > Regards > > > > > Naba > > > > > > > > > > > > > > >
Re: [DISCUSS] - Upgrading from Spring 4 to Spring 5
@Naba, Thank you for this. I think we have identified quite a few PR's (I think we could name around 5 ppl who have started this). The intent is to align the versions with verified BOM versions. THAT should hopefully help this exercise be simplified. --Udo On 11/7/19 3:37 PM, Nabarun Nag wrote: Initial attempts have been made : https://github.com/apache/geode/pull/4256 This PR has Maintainers edit permissions enabled. We need to figure out a plan on these following springframework dependencies too. - spring-hateos [geode - 0.25.0 RELEASE latest 1.0.1 RELEASE] - spring-plugin-core [geode - 1.2.0 RELEASE latest 2.0.0 RELEASE] - spring-plugin-metadata [geode - 1.2.0 RELEASE latest 2.0.0 RELEASE] On Thu, Nov 7, 2019 at 2:43 PM Jason Huynh wrote: +1 On Thu, Nov 7, 2019 at 1:28 PM Dan Smith wrote: +1 On Thu, Nov 7, 2019 at 12:49 PM Jens Deppe wrote: +1 On Wed, Oct 30, 2019 at 1:39 PM Udo Kohlmeyer wrote: Sorry, To clarify... When we change the Spring version we would be looking at looking to use the latest version and it's associated BOM. That might be inclusive of other Spring project upgrades. --Udo On 10/30/19 1:35 PM, Nabarun Nag wrote: Hi Udo, Maven has the latest as 5.2.0.RELEASE as the latest version. In the Dependency.groovy file, we have been putting the full version number. Hence I am guessing you are suggesting we put 5.2.0.RELEASE? What about the status of the following dependencies? 'org.springframework.hateoas', name: 'spring-hateoas', version: '0.25.0.RELEASE' 'org.springframework.ldap', name: 'spring-ldap-core', version: '2.3.2.RELEASE' 'org.springframework.shell', name: 'spring-shell', version: '1.2.0.RELEASE' Regards Naba
Concourse `geode-build-version` is not being incremented
Sometime around when we created a release branch for v1.11.0, and bumped develop to v1.12.0, the semver resource stopped incrementing our SNAPSHOT number. The 1.11 main pipeline is stuck at 1.11.0 (the initial version) instead of having continued on from 1.11.0-SNAPSHOT.303 The develop main pipeline is stuck at 1.12.0, after having briefly been at 1.12.0-SNAPSHOT.1 Something fishy is going on. And I cannot determine if it is related to the branching of geode, or to the roll of Concourse from 5.6.0 to 5.7.0. Does anyone have any guesses? Should we pause both pipelines until someone can sort this out? -Robert
