Re: failing: not ok 3 auth enable/disable lifecycle in 10000ms

2024-01-22 Thread Eric Pugh
Build https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/168/ has been 
running for five days!   Guessing it's not going to finish.

I clicked the "Stop" button on the build and will see what happens.  I might 
let it run another time to see if the test starts behaving, and if not, then 
remove the test.   I think this needs to get resolved one way or the other 
before 9.5 gets cut!

Eric


On 2024/01/18 15:00:40 Eric Pugh wrote:
> Okay, I checked and it looks like the test failed, and is causing the entire 
> build to hang...  
> 
> https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/168/console
> 
> The test runs on my local machine just fineAnd it appears to be 
> running just fine on Solr-Check-main.   
> 
> Is there a way to kill and restart the jenkins build and see if it happens 
> again?   Alternatively I could live with just removing this bats test from 
> the branch_9x?
> 
> On 2024/01/17 19:35:26 Eric Pugh wrote:
> > I hope I did the right thing, which was I fixed the back port by making a 
> > change and committing directly to branch_9x.   
> > 
> > 
> > https://github.com/apache/solr/commit/acaed85f8544b66b127175df02c798e74dd90ff2
> > Fix SOLR-16876 backport of bin/solr scripts to work with 9x SolrCLI.java · 
> > apache/solr@acaed85
> > github.com
> > 
> > 
> > I will keep an eye on the Solr-Check-9.x build.
> > 
> > 
> > 
> > > On Jan 17, 2024, at 11:05 AM, Eric Pugh  
> > > wrote:
> > > 
> > > I will take a look….
> > > 
> > >> On Jan 17, 2024, at 10:38 AM, David Smiley  wrote:
> > >> 
> > >> The integration tests (bats) are failing on branch_9x for me locally and 
> > >> as
> > >> also seen in Jenkins.
> > >>> not ok 3 auth enable/disable lifecycle in 1ms
> > >> https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/167/console
> > >> 
> > >> Hopefully someone has a clue why?
> > >> 
> > >> ~ David Smiley
> > >> Apache Lucene/Solr Search Developer
> > >> http://www.linkedin.com/in/davidwsmiley
> > > 
> > > ___
> > > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> > > http://www.opensourceconnections.com 
> > >  | My Free/Busy 
> > >   
> > > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 
> > > 
> > >   
> > > This e-mail and all contents, including attachments, is considered to be 
> > > Company Confidential unless explicitly stated otherwise, regardless of 
> > > whether attachments are marked as such.
> > > 
> > 
> > ___
> > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
> > http://www.opensourceconnections.com 
> >  | My Free/Busy 
> >   
> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 
> > 
> > 
> > This e-mail and all contents, including attachments, is considered to be 
> > Company Confidential unless explicitly stated otherwise, regardless of 
> > whether attachments are marked as such.
> > 
> > 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> For additional commands, e-mail: dev-h...@solr.apache.org
> 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org



Protected-Branches Maintenance on Release

2024-01-22 Thread Jason Gerlowski
Hey all,

I noticed some instructions in the releaseWizard that seemed a little "off"
around how protected branches are updated.  Particularly, it looks like the
wizard has RMs update ".asf.yaml" on the new release branch (e.g.
branch_9_5), instead of 'main'.

(The relevant wizard code is here:
https://github.com/apache/solr/blob/main/dev-tools/scripts/releaseWizard.yaml#L399-L428
)

I suspect that this change to "asf.yaml" should really be happening on
'main' instead, but haven't been able to find an answer in the ASF docs to
confirm this one way or another.  Can anyone that knows more about the
ASF-Github integration confirm or trash my theory here?

Best,

Jason


Re: Protected-Branches Maintenance on Release

2024-01-22 Thread Jason Gerlowski
Relatedly, there's a PR here to update the file on 'main':
https://github.com/apache/solr/pull/2211

On Mon, Jan 22, 2024 at 11:08 AM Jason Gerlowski 
wrote:

> Hey all,
>
> I noticed some instructions in the releaseWizard that seemed a little
> "off" around how protected branches are updated.  Particularly, it looks
> like the wizard has RMs update ".asf.yaml" on the new release branch (e.g.
> branch_9_5), instead of 'main'.
>
> (The relevant wizard code is here:
> https://github.com/apache/solr/blob/main/dev-tools/scripts/releaseWizard.yaml#L399-L428
> )
>
> I suspect that this change to "asf.yaml" should really be happening on
> 'main' instead, but haven't been able to find an answer in the ASF docs to
> confirm this one way or another.  Can anyone that knows more about the
> ASF-Github integration confirm or trash my theory here?
>
> Best,
>
> Jason
>


New branch and feature freeze for Solr 9.5.0

2024-01-22 Thread Jason Gerlowski
NOTICE:

Branch branch_9_5 has been cut and versions updated to 9.6 on the stable
branch.

Please observe the normal rules:

* No new features may be committed to the branch.
* Documentation patches, build patches and serious bug fixes may be
  committed to the branch. However, you should submit all patches you
  want to commit to Jira first to give others the chance to review
  and possibly vote against the patch. Keep in mind that it is our
  main intention to keep the branch as stable as possible.
* All patches that are intended for the branch should first be committed
  to the unstable branch, merged into the stable branch, and then into
  the current release branch.
* Normal unstable and stable branch development may continue as usual.
  However, if you plan to commit a big change to the unstable branch
  while the branch feature freeze is in effect, think twice: can't the
  addition wait a couple more days? Merges of bug fixes into the branch
  may become more difficult.
* Only Jira issues with Fix version 9.5 and priority "Blocker" will delay
  a release candidate build.

The feature-freeze for the 9.5 release will go till the end of this week -
I'll aim to create our first RC on Thursday, January 25th.

Best,

Jason


Re: [DISCUSS] Solr 9.5 Release

2024-01-22 Thread Jason Gerlowski
Hey all,

You've likely seen the "New branch and feature freeze for Solr 9.5.0"
email, but in case you haven't:

branch_9_5 is now created and "feature freeze" for it is currently in
effect.  There are no 9.5 blockers afaik, so my plan is to leave a few days
for any lingering bugs folks might want to squeeze in and then try for an
RC this Thursday.

Best,

Jason

On Fri, Jan 19, 2024 at 1:29 PM Jason Gerlowski 
wrote:

> Good question - I guess this is now "unblocked" now that 9.4.1 is out the
> door (thanks David!).
>
> I guess, pending any objections, that my revised timing would be to begin
> "feature freeze" and create branch_9_5 Monday or Tuesday of next week.
> With the hope that I can get the first RC out by the end of the week?
>
> Best,
>
> Jason
>
> On Fri, Jan 19, 2024 at 12:40 PM David Smiley  wrote:
>
>> When is the feature freeze?  Please create the branch_9_5 at the time of
>> that.
>>
>> On Tue, Jan 16, 2024 at 3:45 PM Jason Gerlowski 
>> wrote:
>>
>> > > Is there an urgent need for this release?
>> >
>> > Have I seemed "urgent"?  I suggested a mid-January 9.5 back on 12/6 (see
>> > the "9.4.1 release" thread).  If anything I've been slow :-p
>> >
>> > Like I said in my initial message here - my motivation around 9.5 is
>> just
>> > that I think there's some cool stuff lined up and I'd love to get that
>> out
>> > to folks.  Getting features out is a "Good Thing", so why wait on 9.5 if
>> > there's not a particular reason to?  If there is a concrete reason to
>> wait,
>> > that's fine too; I'm just curious to learn more about it.
>> >
>> > Best,
>> >
>> > Jason
>> >
>> >
>> >
>> > On Tue, Jan 16, 2024 at 1:50 PM Houston Putman 
>> wrote:
>> >
>> > > Apple isn't urging anything, and you really can assume that's true
>> across
>> > > the board for our decisions.
>> > >
>> > > There are other concerns that are urging quick releases (See other
>> > mailing
>> > > lists for more information).
>> > >
>> > > I'm happy to let you continue the 8.11.3 release, and if you want to
>> wait
>> > > for 9.4.1 that is fine with me. But we do need to get it out quickly.
>> > >
>> > > - Houston
>> > >
>> > > On Tue, Jan 16, 2024 at 12:45 PM Ishan Chattopadhyaya <
>> > > ichattopadhy...@gmail.com> wrote:
>> > >
>> > > > Is there an urgent need for this release?
>> > > > I saw that Houston took over the 8.11.3 release process from me
>> without
>> > > any
>> > > > prior intimation. Is Apple asking both of you to hurry up?
>> > > >
>> > > > On Tue, 16 Jan 2024 at 20:35, Jason Gerlowski <
>> gerlowsk...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > > but let 9.4.1 get out the door first..
>> > > > >
>> > > > > I'm happy to wait if that's the consensus, but is there a reason
>> one
>> > > > > release needs to block another?
>> > > > >
>> > > > > I could be blanking, but I don't think there's any technical
>> conflict
>> > > in
>> > > > > terms of where artifacts are staged, etc. is there?
>> > > > >
>> > > > > > getting through some one-time pains of key/GPG matters
>> > > > >
>> > > > > Definitely the worst bit in my recent experience; good luck!
>> > > > >
>> > > > > Best,
>> > > > >
>> > > > > Jason
>> > > > >
>> > > > > On Fri, Jan 12, 2024 at 11:08 PM David Smiley > >
>> > > > wrote:
>> > > > >
>> > > > > > Yeah I'm doing the 9.4.1 for real now... getting through some
>> > > one-time
>> > > > > > pains of key/GPG matters.
>> > > > > >
>> > > > > > https://issues.apache.org/jira/browse/SOLR-17096 solr.xml
>> support
>> > > for
>> > > > > >  plugins is pretty close!
>> > > > > >
>> > > > > > On Fri, Jan 12, 2024 at 3:56 PM Jason Gerlowski <
>> > > gerlowsk...@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hey all,
>> > > > > > >
>> > > > > > > branch_9x has accumulated 3 new features, 19 improvements, 2
>> > > > > > optimizations,
>> > > > > > > 11 bug fixes, and 7 "other changes" - I'd love to get these in
>> > > front
>> > > > of
>> > > > > > > users with a Solr 9.5.0 release.
>> > > > > > >
>> > > > > > > I'm happy to volunteer as RM, and propose that we target next
>> > > > Thursday
>> > > > > > > January 18th to cut the branch and prepare the first RC.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > >
>> > > > > > > Jason
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>


Re: New branch and feature freeze for Solr 9.5.0

2024-01-22 Thread Houston Putman
Hey Jason,

I committed a bug in the build for setting up Node with a registry mirror.
I assume it's ok for me to backport?

Also I noticed that there is an issue with the ref guide. One page has two
of the same anchors on branch_9x and branch_9_5. We should fix this before
the release.

{"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":606},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
> assigned to block already in use: v1coreadmin-mergeindexes"}
> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":614},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
> assigned to block already in use: v2coreadmin-mergeindexes"}
>

- Houston

On Mon, Jan 22, 2024 at 12:30 PM Jason Gerlowski 
wrote:

> NOTICE:
>
> Branch branch_9_5 has been cut and versions updated to 9.6 on the stable
> branch.
>
> Please observe the normal rules:
>
> * No new features may be committed to the branch.
> * Documentation patches, build patches and serious bug fixes may be
>   committed to the branch. However, you should submit all patches you
>   want to commit to Jira first to give others the chance to review
>   and possibly vote against the patch. Keep in mind that it is our
>   main intention to keep the branch as stable as possible.
> * All patches that are intended for the branch should first be committed
>   to the unstable branch, merged into the stable branch, and then into
>   the current release branch.
> * Normal unstable and stable branch development may continue as usual.
>   However, if you plan to commit a big change to the unstable branch
>   while the branch feature freeze is in effect, think twice: can't the
>   addition wait a couple more days? Merges of bug fixes into the branch
>   may become more difficult.
> * Only Jira issues with Fix version 9.5 and priority "Blocker" will delay
>   a release candidate build.
>
> The feature-freeze for the 9.5 release will go till the end of this week -
> I'll aim to create our first RC on Thursday, January 25th.
>
> Best,
>
> Jason
>


Re: New branch and feature freeze for Solr 9.5.0

2024-01-22 Thread Jason Gerlowski
> I committed a bug in the build for setting up Node with a registry
mirror.  I assume it's ok for me to backport?

Yep, please do!

> One page has two of the same anchors on branch_9x and branch_9_5. We
should fix this before the release.

Agreed, I'll take a look.

Fwiw, I merged the PR that likely created that anchor conflict.  I'm a
little surprised that this isn't caught by the 'check'/'precommit' checks -
I've been relying (apparently mistakenly) on those tasks to catch this sort
of thing when reviewing and merging PRs.  I guess I'll have to add some
other ref-guide specific task to my pre-merge routine?

On Mon, Jan 22, 2024 at 1:29 PM Houston Putman  wrote:

> Hey Jason,
>
> I committed a bug in the build for setting up Node with a registry mirror.
> I assume it's ok for me to backport?
>
> Also I noticed that there is an issue with the ref guide. One page has two
> of the same anchors on branch_9x and branch_9_5. We should fix this before
> the release.
>
>
> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":606},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
> > assigned to block already in use: v1coreadmin-mergeindexes"}
> >
> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":614},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
> > assigned to block already in use: v2coreadmin-mergeindexes"}
> >
>
> - Houston
>
> On Mon, Jan 22, 2024 at 12:30 PM Jason Gerlowski 
> wrote:
>
> > NOTICE:
> >
> > Branch branch_9_5 has been cut and versions updated to 9.6 on the stable
> > branch.
> >
> > Please observe the normal rules:
> >
> > * No new features may be committed to the branch.
> > * Documentation patches, build patches and serious bug fixes may be
> >   committed to the branch. However, you should submit all patches you
> >   want to commit to Jira first to give others the chance to review
> >   and possibly vote against the patch. Keep in mind that it is our
> >   main intention to keep the branch as stable as possible.
> > * All patches that are intended for the branch should first be committed
> >   to the unstable branch, merged into the stable branch, and then into
> >   the current release branch.
> > * Normal unstable and stable branch development may continue as usual.
> >   However, if you plan to commit a big change to the unstable branch
> >   while the branch feature freeze is in effect, think twice: can't the
> >   addition wait a couple more days? Merges of bug fixes into the branch
> >   may become more difficult.
> > * Only Jira issues with Fix version 9.5 and priority "Blocker" will delay
> >   a release candidate build.
> >
> > The feature-freeze for the 9.5 release will go till the end of this week
> -
> > I'll aim to create our first RC on Thursday, January 25th.
> >
> > Best,
> >
> > Jason
> >
>


Re: New branch and feature freeze for Solr 9.5.0

2024-01-22 Thread Jason Gerlowski
I've put together a first-draft for our release-notes for 9.5.  They're
available here:
https://cwiki.apache.org/confluence/display/SOLR/DRAFT+ReleaseNote9_5_0

Would love some review on what I've put together; feel free to add, remove
or change anything as needed!

Best,

Jason

On Mon, Jan 22, 2024 at 2:18 PM Jason Gerlowski 
wrote:

> > I committed a bug in the build for setting up Node with a registry
> mirror.  I assume it's ok for me to backport?
>
> Yep, please do!
>
> > One page has two of the same anchors on branch_9x and branch_9_5. We
> should fix this before the release.
>
> Agreed, I'll take a look.
>
> Fwiw, I merged the PR that likely created that anchor conflict.  I'm a
> little surprised that this isn't caught by the 'check'/'precommit' checks -
> I've been relying (apparently mistakenly) on those tasks to catch this sort
> of thing when reviewing and merging PRs.  I guess I'll have to add some
> other ref-guide specific task to my pre-merge routine?
>
> On Mon, Jan 22, 2024 at 1:29 PM Houston Putman  wrote:
>
>> Hey Jason,
>>
>> I committed a bug in the build for setting up Node with a registry mirror.
>> I assume it's ok for me to backport?
>>
>> Also I noticed that there is an issue with the ref guide. One page has two
>> of the same anchors on branch_9x and branch_9_5. We should fix this before
>> the release.
>>
>>
>> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":606},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
>> > assigned to block already in use: v1coreadmin-mergeindexes"}
>> >
>> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":614},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
>> > assigned to block already in use: v2coreadmin-mergeindexes"}
>> >
>>
>> - Houston
>>
>> On Mon, Jan 22, 2024 at 12:30 PM Jason Gerlowski 
>> wrote:
>>
>> > NOTICE:
>> >
>> > Branch branch_9_5 has been cut and versions updated to 9.6 on the stable
>> > branch.
>> >
>> > Please observe the normal rules:
>> >
>> > * No new features may be committed to the branch.
>> > * Documentation patches, build patches and serious bug fixes may be
>> >   committed to the branch. However, you should submit all patches you
>> >   want to commit to Jira first to give others the chance to review
>> >   and possibly vote against the patch. Keep in mind that it is our
>> >   main intention to keep the branch as stable as possible.
>> > * All patches that are intended for the branch should first be committed
>> >   to the unstable branch, merged into the stable branch, and then into
>> >   the current release branch.
>> > * Normal unstable and stable branch development may continue as usual.
>> >   However, if you plan to commit a big change to the unstable branch
>> >   while the branch feature freeze is in effect, think twice: can't the
>> >   addition wait a couple more days? Merges of bug fixes into the branch
>> >   may become more difficult.
>> > * Only Jira issues with Fix version 9.5 and priority "Blocker" will
>> delay
>> >   a release candidate build.
>> >
>> > The feature-freeze for the 9.5 release will go till the end of this
>> week -
>> > I'll aim to create our first RC on Thursday, January 25th.
>> >
>> > Best,
>> >
>> > Jason
>> >
>>
>


Re: New branch and feature freeze for Solr 9.5.0

2024-01-22 Thread Jason Gerlowski
> I'm a little surprised that this isn't caught by the
'check'/'precommit' checks - I've been relying (apparently mistakenly) on
those tasks to catch this sort of thing when reviewing and merging PRs.

Ah, I see why I didn't catch it.  It's a warning instead of an error.
Nevermind my comment.  Working on a fix now.

On Mon, Jan 22, 2024 at 2:32 PM Jason Gerlowski 
wrote:

> I've put together a first-draft for our release-notes for 9.5.  They're
> available here:
> https://cwiki.apache.org/confluence/display/SOLR/DRAFT+ReleaseNote9_5_0
>
> Would love some review on what I've put together; feel free to add, remove
> or change anything as needed!
>
> Best,
>
> Jason
>
> On Mon, Jan 22, 2024 at 2:18 PM Jason Gerlowski 
> wrote:
>
>> > I committed a bug in the build for setting up Node with a registry
>> mirror.  I assume it's ok for me to backport?
>>
>> Yep, please do!
>>
>> > One page has two of the same anchors on branch_9x and branch_9_5. We
>> should fix this before the release.
>>
>> Agreed, I'll take a look.
>>
>> Fwiw, I merged the PR that likely created that anchor conflict.  I'm a
>> little surprised that this isn't caught by the 'check'/'precommit' checks -
>> I've been relying (apparently mistakenly) on those tasks to catch this sort
>> of thing when reviewing and merging PRs.  I guess I'll have to add some
>> other ref-guide specific task to my pre-merge routine?
>>
>> On Mon, Jan 22, 2024 at 1:29 PM Houston Putman 
>> wrote:
>>
>>> Hey Jason,
>>>
>>> I committed a bug in the build for setting up Node with a registry
>>> mirror.
>>> I assume it's ok for me to backport?
>>>
>>> Also I noticed that there is an issue with the ref guide. One page has
>>> two
>>> of the same anchors on branch_9x and branch_9_5. We should fix this
>>> before
>>> the release.
>>>
>>>
>>> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":606},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
>>> > assigned to block already in use: v1coreadmin-mergeindexes"}
>>> >
>>> {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":614},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
>>> > assigned to block already in use: v2coreadmin-mergeindexes"}
>>> >
>>>
>>> - Houston
>>>
>>> On Mon, Jan 22, 2024 at 12:30 PM Jason Gerlowski 
>>> wrote:
>>>
>>> > NOTICE:
>>> >
>>> > Branch branch_9_5 has been cut and versions updated to 9.6 on the
>>> stable
>>> > branch.
>>> >
>>> > Please observe the normal rules:
>>> >
>>> > * No new features may be committed to the branch.
>>> > * Documentation patches, build patches and serious bug fixes may be
>>> >   committed to the branch. However, you should submit all patches you
>>> >   want to commit to Jira first to give others the chance to review
>>> >   and possibly vote against the patch. Keep in mind that it is our
>>> >   main intention to keep the branch as stable as possible.
>>> > * All patches that are intended for the branch should first be
>>> committed
>>> >   to the unstable branch, merged into the stable branch, and then into
>>> >   the current release branch.
>>> > * Normal unstable and stable branch development may continue as usual.
>>> >   However, if you plan to commit a big change to the unstable branch
>>> >   while the branch feature freeze is in effect, think twice: can't the
>>> >   addition wait a couple more days? Merges of bug fixes into the branch
>>> >   may become more difficult.
>>> > * Only Jira issues with Fix version 9.5 and priority "Blocker" will
>>> delay
>>> >   a release candidate build.
>>> >
>>> > The feature-freeze for the 9.5 release will go till the end of this
>>> week -
>>> > I'll aim to create our first RC on Thursday, January 25th.
>>> >
>>> > Best,
>>> >
>>> > Jason
>>> >
>>>
>>


Re: New branch and feature freeze for Solr 9.5.0

2024-01-22 Thread Eric Pugh
Should it be an error?

> On Jan 22, 2024, at 2:35 PM, Jason Gerlowski  wrote:
> 
>> I'm a little surprised that this isn't caught by the
> 'check'/'precommit' checks - I've been relying (apparently mistakenly) on
> those tasks to catch this sort of thing when reviewing and merging PRs.
> 
> Ah, I see why I didn't catch it.  It's a warning instead of an error.
> Nevermind my comment.  Working on a fix now.
> 
> On Mon, Jan 22, 2024 at 2:32 PM Jason Gerlowski 
> wrote:
> 
>> I've put together a first-draft for our release-notes for 9.5.  They're
>> available here:
>> https://cwiki.apache.org/confluence/display/SOLR/DRAFT+ReleaseNote9_5_0
>> 
>> Would love some review on what I've put together; feel free to add, remove
>> or change anything as needed!
>> 
>> Best,
>> 
>> Jason
>> 
>> On Mon, Jan 22, 2024 at 2:18 PM Jason Gerlowski 
>> wrote:
>> 
 I committed a bug in the build for setting up Node with a registry
>>> mirror.  I assume it's ok for me to backport?
>>> 
>>> Yep, please do!
>>> 
 One page has two of the same anchors on branch_9x and branch_9_5. We
>>> should fix this before the release.
>>> 
>>> Agreed, I'll take a look.
>>> 
>>> Fwiw, I merged the PR that likely created that anchor conflict.  I'm a
>>> little surprised that this isn't caught by the 'check'/'precommit' checks -
>>> I've been relying (apparently mistakenly) on those tasks to catch this sort
>>> of thing when reviewing and merging PRs.  I guess I'll have to add some
>>> other ref-guide specific task to my pre-merge routine?
>>> 
>>> On Mon, Jan 22, 2024 at 1:29 PM Houston Putman 
>>> wrote:
>>> 
 Hey Jason,
 
 I committed a bug in the build for setting up Node with a registry
 mirror.
 I assume it's ok for me to backport?
 
 Also I noticed that there is an issue with the ref guide. One page has
 two
 of the same anchors on branch_9x and branch_9_5. We should fix this
 before
 the release.
 
 
 {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":606},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
> assigned to block already in use: v1coreadmin-mergeindexes"}
> 
 {"level":"warn","time":1705948125542,"name":"asciidoctor","file":{"path":"/Users/houstonputman/dev/oss/solr/solr/main/solr/solr-ref-guide/build/site-staging/modules/configuration-guide/pages/coreadmin-api.adoc","line":614},"source":{"url":"file:///Users/houstonputman/dev/oss/solr/solr/main","local":"/Users/houstonputman/dev/oss/solr/solr/main/.git","worktree":"/Users/houstonputman/dev/oss/solr/solr/main","refname":"branch_9_5","reftype":"branch","startPath":"solr/solr-ref-guide/build/site-staging"},"msg":"id
> assigned to block already in use: v2coreadmin-mergeindexes"}
> 
 
 - Houston
 
 On Mon, Jan 22, 2024 at 12:30 PM Jason Gerlowski 
 wrote:
 
> NOTICE:
> 
> Branch branch_9_5 has been cut and versions updated to 9.6 on the
 stable
> branch.
> 
> Please observe the normal rules:
> 
> * No new features may be committed to the branch.
> * Documentation patches, build patches and serious bug fixes may be
>  committed to the branch. However, you should submit all patches you
>  want to commit to Jira first to give others the chance to review
>  and possibly vote against the patch. Keep in mind that it is our
>  main intention to keep the branch as stable as possible.
> * All patches that are intended for the branch should first be
 committed
>  to the unstable branch, merged into the stable branch, and then into
>  the current release branch.
> * Normal unstable and stable branch development may continue as usual.
>  However, if you plan to commit a big change to the unstable branch
>  while the branch feature freeze is in effect, think twice: can't the
>  addition wait a couple more days? Merges of bug fixes into the branch
>  may become more difficult.
> * Only Jira issues with Fix version 9.5 and priority "Blocker" will
 delay
>  a release candidate build.
> 
> The feature-freeze for the 9.5 release will go till the end of this
 week -
> I'll aim to create our first RC on Thursday, January 25th.
> 
> Best,
> 
> Jason
> 
 
>>> 

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-22 Thread Jason Gerlowski
Thanks for the detailed response!  I've got a few follow-ups that I'll ask
in line, but I think I've got the core idea now, thanks!

> With the implementation in its current state, an indexing batch (as sent
> by the client) is processed by the leader replica, committed to local disk
> then written into the remote storage (S3) before SolrCloud acknowledges
the
>  batch. Transaction log is not used and the commit is forced.

To clarify - since you mentioned durability, I'm assuming the commit that
happens on each update batch is a "hard commit"?

If the user made an explicit hard-commit or had an autoCommit configured,
would that be a no-op for ZERO collections?

How do soft-commits behave - I imagine they're handled similarly to
whatever is done for PULL replicas currently?

> Currently the (basic) approach is that a replica checks if it is up to
> date as long as it's getting queries. If it is already up to date, the
cost
> of that check is a ZooKeeper node read.

And that ZK read happens on each query?  Or does the current strategy have
it check "every X seconds" like how PULL/TLOG replicas check for updates?

Is that ZK check looking at the "shard term" stuff, or would Zero
replicas/collections store some additional state in ZK to manage this?

Best,

Jason

On Wed, Jan 17, 2024 at 8:05 PM Ilan Ginzburg  wrote:

> Hi Jason,
>
> Good questions!
> Note there is a related question on the users list, see thread Node roles
> vs SIP-20 Separation of Compute and Storage
> 
>
> When I share the code (later this week in all likelihood) I will share a
> detailed write up. I'm open to discuss/present/write more as needed (and
> plan to attend Thursday 18/1 Meetup
> 
> to
> present the SIP).
>
> To your questions:
>
>- The name "Zero" is because "Shared" was too broad (and already used)
>and "Blob" was already used as well in Solr code, and we didn't find any
>better name (but the name can be changed, there are a couple thousand
>occurrences of it in different forms in the code, I know because "Zero"
> is
>not the name we use internally at Salesforce for that replica type, I've
>renamed everything for sharing here). The reason for "Zero" is that
> there
>is only one instance (zero copies) in persistent storage for each shard,
>and also this evoques the (longer term) option to have zero (i.e. no)
>replicas for a shard on the cluster, pick a node, load and materialize a
>replica when needed. But that's longer term.
>- Your understanding is correct, currently it's a kind of new type of
>collection given all replicas are ZERO (or none are) for a collection.
> ZERO
>could have been a new type of shard, as one could imagine different
> shards
>having different types (but shards do not have types), or why not also
>allow a given shard to have ZERO replicas as well as other types of
>replicas (this is not supported in the current implementation for
>simplicity and we didn't really felt the need). If collectively we think
>that PULL + ZERO do make sense, why not. PULL would then be fetching its
>content from ZERO replicas rather than directly from the shared
> storage. I
>don't see ZERO coexisting with NRT or TLOG in a given shard though.
>Currently as the implementation forces all replicas of all shards of a
>"Zero" collection to be ZERO, there is a flag on that collection saying
> it
>is a Zero collection.
>- How the compute behaves:
>With the implementation in its current state, an indexing batch (as sent
>by the client) is processed by the leader replica, committed to local
> disk
>then written into the remote storage (S3) before SolrCloud acknowledges
> the
>batch. Transaction log is not used and the commit is forced. This slows
>down indexing, but a design guideline from the start was that the
> SolrCloud
>nodes are stateless (can restart with empty disk) to simplify managing
>elasticity (and in a public cloud setting, Availability Zone failures).
>The next evolution step of this code that we plan to start upstream once
>the branch has been shared is to enable the transaction log for ZERO
>replicas, and have indexing behave more like a normal SolrCloud: persist
>the transaction log before replying to a client, do not force a commit.
> The
>transaction log would be changed to be on shared storage as well, with a
>single copy for a shard, not one log per node/replica. All replicas of
> the
>shard, when they're becoming leaders, would access the same transaction
>log. Some of the logic used for implementing the shared storage for the
>index segments will be reused.
>For serving queries, the non leader replicas (also of type ZERO) update
>themselves from the shared storage directly. They behave mostly like
> PULL
>

Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-22 Thread Ilan Ginzburg
>
> To clarify - since you mentioned durability, I'm assuming the commit that
> happens on each update batch is a "hard commit"?
>
Yes.

If the user made an explicit hard-commit or had an autoCommit configured,
> would that be a no-op for ZERO collections?
>
Yes.


> How do soft-commits behave - I imagine they're handled similarly to
> whatever is done for PULL replicas currently?
>
Currently a hard commit is automatically added after the last document of a
batch. Not sure what would happen if that wasn't the case and a soft commit
was issued instead.
In any case the soft commit would not propagate to other ZERO replicas than
the leader that's for sure (I assume when you mention soft commit and PULL
you mean soft commit on another replica?). The soft commit will also not
write the files to the shared repository. So likely the batch would be
acknowledged to the client but would be lost if a failure hit the node
before the next (hard) commit (no transaction log, nodes are assumed
stateless, so Zero replicas skip recovery on node startup).

And that ZK read happens on each query?  Or does the current strategy have
> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>
(about a replica checking with a ZK read if it's up to date). In the
current implementation such a read is enqueued each time there's a query,
but basically until that read has happened then another query arrived,
there will not be a second read. So if for example a lot of queries arrive
in the same second before the read happens, there would be a single read.
Or if the cluster is loaded (many replicas get queries and enqueue such
reads from ZK), the reads will not be executed right away and no new read
will be enqueued until the previous one was done.
Spacing such reads by some minimum delay is not a big change.

Is that ZK check looking at the "shard term" stuff, or would Zero
> replicas/collections store some additional state in ZK to manage this?

 It is not shard terms. Terms are being used when a follower replica does
not manage to get an indexing update from the leader. Given Zero replicas
do not send indexing updates to other replicas, terms are never increased
and are therefore not used for this type of replica.
The value that is read in ZK is basically the pointer to the current
metadata file for the shard in the [Backup]Repository (it's called
BackupRepository but the Zero implementation uses it as a general purpose
repository). When the shard in the repository changes, the shard metadata
file changes (you can think of it as listing the current stored commit
point in the repository). The shard ZooKeeper node references the current
valid shard metadata file in the repository. A node that has read that
value (then the repository metadata file then all the segments) can check
if the ZooKeeper node has changed, and if it has not, the node already has
the current version of the shard.

Ilan

On Mon, Jan 22, 2024 at 10:20 PM Jason Gerlowski 
wrote:

> Thanks for the detailed response!  I've got a few follow-ups that I'll ask
> in line, but I think I've got the core idea now, thanks!
>
> > With the implementation in its current state, an indexing batch (as sent
> > by the client) is processed by the leader replica, committed to local
> disk
> > then written into the remote storage (S3) before SolrCloud acknowledges
> the
> >  batch. Transaction log is not used and the commit is forced.
>
> To clarify - since you mentioned durability, I'm assuming the commit that
> happens on each update batch is a "hard commit"?
>
> If the user made an explicit hard-commit or had an autoCommit configured,
> would that be a no-op for ZERO collections?
>
> How do soft-commits behave - I imagine they're handled similarly to
> whatever is done for PULL replicas currently?
>
> > Currently the (basic) approach is that a replica checks if it is up to
> > date as long as it's getting queries. If it is already up to date, the
> cost
> > of that check is a ZooKeeper node read.
>
> And that ZK read happens on each query?  Or does the current strategy have
> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>
> Is that ZK check looking at the "shard term" stuff, or would Zero
> replicas/collections store some additional state in ZK to manage this?
>
> Best,
>
> Jason
>
> On Wed, Jan 17, 2024 at 8:05 PM Ilan Ginzburg  wrote:
>
> > Hi Jason,
> >
> > Good questions!
> > Note there is a related question on the users list, see thread Node roles
> > vs SIP-20 Separation of Compute and Storage
> > 
> >
> > When I share the code (later this week in all likelihood) I will share a
> > detailed write up. I'm open to discuss/present/write more as needed (and
> > plan to attend Thursday 18/1 Meetup
> > <
> https://cwiki.apache.org/confluence/display/SOLR/2024-01-18+Meeting+notes>
> > to
> > present the SIP).
> >
> > To your questions:
> >
> >- The name "Zero" is becau

Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-22 Thread Ilan Ginzburg
I have created a Jira, the branch with the current code and
documentation and updated the SIP-20

 page:

Jira: SOLR-17125 
Branch: jira/solr-17125-zero-replicas

Documentation (in the branch): ZERO-REPLICAS.adoc


This code is based on code running in production at large scale, but that
code underwent heavy cleanup, preparation and porting for sharing it here.
It is quite possible that regressions were introduced, the code as
presented *did not* run in production.

I'd be happy to answer any questions and if there's interest, I can host
live sessions for code walkthrough or discussions.

Ilan


On Tue, Jan 23, 2024 at 1:49 AM Ilan Ginzburg  wrote:

> To clarify - since you mentioned durability, I'm assuming the commit that
>> happens on each update batch is a "hard commit"?
>>
> Yes.
>
> If the user made an explicit hard-commit or had an autoCommit configured,
>> would that be a no-op for ZERO collections?
>>
> Yes.
>
>
>> How do soft-commits behave - I imagine they're handled similarly to
>> whatever is done for PULL replicas currently?
>>
> Currently a hard commit is automatically added after the last document of
> a batch. Not sure what would happen if that wasn't the case and a soft
> commit was issued instead.
> In any case the soft commit would not propagate to other ZERO replicas
> than the leader that's for sure (I assume when you mention soft commit and
> PULL you mean soft commit on another replica?). The soft commit will also
> not write the files to the shared repository. So likely the batch would be
> acknowledged to the client but would be lost if a failure hit the node
> before the next (hard) commit (no transaction log, nodes are assumed
> stateless, so Zero replicas skip recovery on node startup).
>
> And that ZK read happens on each query?  Or does the current strategy have
>> it check "every X seconds" like how PULL/TLOG replicas check for updates?
>>
> (about a replica checking with a ZK read if it's up to date). In the
> current implementation such a read is enqueued each time there's a query,
> but basically until that read has happened then another query arrived,
> there will not be a second read. So if for example a lot of queries arrive
> in the same second before the read happens, there would be a single read.
> Or if the cluster is loaded (many replicas get queries and enqueue such
> reads from ZK), the reads will not be executed right away and no new read
> will be enqueued until the previous one was done.
> Spacing such reads by some minimum delay is not a big change.
>
> Is that ZK check looking at the "shard term" stuff, or would Zero
>> replicas/collections store some additional state in ZK to manage this?
>
>  It is not shard terms. Terms are being used when a follower replica does
> not manage to get an indexing update from the leader. Given Zero replicas
> do not send indexing updates to other replicas, terms are never increased
> and are therefore not used for this type of replica.
> The value that is read in ZK is basically the pointer to the current
> metadata file for the shard in the [Backup]Repository (it's called
> BackupRepository but the Zero implementation uses it as a general purpose
> repository). When the shard in the repository changes, the shard metadata
> file changes (you can think of it as listing the current stored commit
> point in the repository). The shard ZooKeeper node references the current
> valid shard metadata file in the repository. A node that has read that
> value (then the repository metadata file then all the segments) can check
> if the ZooKeeper node has changed, and if it has not, the node already has
> the current version of the shard.
>
> Ilan
>
> On Mon, Jan 22, 2024 at 10:20 PM Jason Gerlowski 
> wrote:
>
>> Thanks for the detailed response!  I've got a few follow-ups that I'll ask
>> in line, but I think I've got the core idea now, thanks!
>>
>> > With the implementation in its current state, an indexing batch (as sent
>> > by the client) is processed by the leader replica, committed to local
>> disk
>> > then written into the remote storage (S3) before SolrCloud acknowledges
>> the
>> >  batch. Transaction log is not used and the commit is forced.
>>
>> To clarify - since you mentioned durability, I'm assuming the commit that
>> happens on each update batch is a "hard commit"?
>>
>> If the user made an explicit hard-commit or had an autoCommit configured,
>> would that be a no-op for ZERO collections?
>>
>> How do soft-commits behave - I imagine they're handled similarly to
>> whatever is done for PULL replicas currently?
>>
>> > Currently the (basic) approach is that a replica checks if it is up to
>> > date as long as