Re: Lucene upgrade

2019-11-07 Thread Xiaojian Zhou
Oh, I misunderstood option-1 and option-2. What I vote is Jason's option-1. On Thu, Nov 7, 2019 at 9:19 AM Jason Huynh wrote: > Gester, I don't think we need to write in the old format, we just need the > new format not to be written while old members can potentially read the > lucene files. Op

Re: Lucene upgrade

2019-11-07 Thread Jason Huynh
Gester, I don't think we need to write in the old format, we just need the new format not to be written while old members can potentially read the lucene files. Option 1 can be very similar to Dan's snippet of code. I think Option 2 is going to leave a lot of people unhappy when they get stuck wi

Re: Lucene upgrade

2019-11-06 Thread Xiaojian Zhou
Usually re-creating region and index are expensive and customers are reluctant to do it, according to my memory. We do have an offline reindex scripts or steps (written by Barry?). If that could be an option, they can try that offline tool. I saw from Mario's email, he said: "I didn't found a way

Re: Lucene upgrade

2019-11-06 Thread Jacob Barrett
> On Nov 6, 2019, at 3:36 PM, Jason Huynh wrote: > > Jake - there is a side effect to this in that the user would have to > reimport all their data into the user defined region too. Client apps > would also have to know which of the regions to put into.. also, I may be > misunderstanding this

Re: Lucene upgrade

2019-11-06 Thread Jason Huynh
Dan - LGTM check it in! ;-) (kidding of course) Jake - there is a side effect to this in that the user would have to reimport all their data into the user defined region too. Client apps would also have to know which of the regions to put into.. also, I may be misunderstanding this suggestion, co

Re: Lucene upgrade

2019-11-06 Thread Jacob Barrett
> On Nov 6, 2019, at 2:16 PM, Jason Huynh wrote: > > Jake, -from my understanding, the implementation details of geode-lucene is > that we are using a partitioned region as a "file-system" for lucene > files. Yeah, I didn’t explain well. I mean to say literally create a new region for the n

Re: Lucene upgrade

2019-11-06 Thread Dan Smith
> > 1.) We add some product code/lucene listener to detect whether we have old > versions of geode and if so, do not write to lucene on the newly updated > node until all versions are up to date. Elaborating on this option a little more, this might be as simple as something like the below at the

Re: Lucene upgrade

2019-11-06 Thread Jason Huynh
Jake, -from my understanding, the implementation details of geode-lucene is that we are using a partitioned region as a "file-system" for lucene files. As new servers are rolled, the issue is that the new servers have the new codec. As puts occur on the users data region, the async listeners are

Re: Lucene upgrade

2019-11-06 Thread Xiaojian Zhou
He tried to upgrade lucene version from current 6.6.4 to 8.2. There're some challenges. One challenge is the codec changed, which caused the format of index is also changed. That's why we did not implement it. If he resolved the coding challenges, then rolling upgrade will probably need option-2

Re: Lucene upgrade

2019-11-06 Thread Jacob Barrett
What about “versioning” the region that backs the indexes? Old servers with old license would continue to read/write to old region. New servers would start re-indexing with the new version. Given the async nature of the indexing would the mismatch in indexing for some period of time have an impa

Re: Lucene upgrade

2019-11-06 Thread Jason Huynh
Hi Mario, I think there are a few ways to accomplish what Dan was suggesting...Dan or other's, please chime in with more options/solutions. 1.) We add some product code/lucene listener to detect whether we have old versions of geode and if so, do not write to lucene on the newly updated node unti

Re: Lucene upgrade

2019-11-06 Thread Mario Kevo
Hi Dan, thanks for suggestions. I didn't found a way to write lucene in older format. They only support reading old format indexes with newer version by using lucene-backward- codec. Regarding to freeze writes to the lucene index, that means that we need to start locators and servers, create luce

Re: Lucene upgrade

2019-11-04 Thread Dan Smith
I think the issue probably has to do with doing a rolling upgrade from an old version of geode (with an old version of lucene) to the new version of geode. Geode's lucene integration works by writing the lucene index to a colocated region. So lucene index data that was generated on one server can