Re: [VOTE] Release Solr 9.6.0 RC1
Hi, Smoketester succeeds: SUCCESS! [0:45:28.467115] But I cannot get the docker build to pass: It seems to fail during GPG validation of your key during image build: > gpg: key 140BC45803B03F7F: public key "Patrick Gustav Heck (CODE SIGNING KEY) > " imported > gpg: can't connect to the agent: IPC connect call failed > gpg: Total number processed: 1 > gpg: imported: 1 > Not sure if that is the reason, but some command in the RUN line 30- fail. +0 Jan > 23. apr. 2024 kl. 07:05 skrev Gus Heck : > > Please vote for release candidate 1 for Solr 9.6.0 > > The artifacts can be downloaded from: > https://dist.apache.org/repos/dist/dev/solr/solr-9.6.0-RC1-rev-f8e5a93c11267e13b7b43005a428bfb910ac6e57 > > You can run the smoke tester directly with this command: > > python3 -u dev-tools/scripts/smokeTestRelease.py \ > https://dist.apache.org/repos/dist/dev/solr/solr-9.6.0-RC1-rev-f8e5a93c11267e13b7b43005a428bfb910ac6e57 > > You can build a release-candidate of the official docker images (full & > slim) using the following command: > > SOLR_DOWNLOAD_SERVER= > https://dist.apache.org/repos/dist/dev/solr/solr-9.6.0-RC1-rev-f8e5a93c11267e13b7b43005a428bfb910ac6e57/solr > && \ > docker build $SOLR_DOWNLOAD_SERVER/9.6.0/docker/Dockerfile.official-full \ > --build-arg SOLR_DOWNLOAD_SERVER=$SOLR_DOWNLOAD_SERVER \ > -t solr-rc:9.6.0-1 && \ > docker build $SOLR_DOWNLOAD_SERVER/9.6.0/docker/Dockerfile.official-slim \ > --build-arg SOLR_DOWNLOAD_SERVER=$SOLR_DOWNLOAD_SERVER \ > -t solr-rc:9.6.0-1-slim > > The vote will be open for at least 72 hours i.e. until 2024-04-26 06:00 UTC. > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and reason why) > > Here is my +1 > > > -- > http://www.needhamsoftware.com (work) > https://a.co/d/b2sZLD9 (my fantasy fiction book) - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
Re: [VOTE] Release Solr 9.6.0 RC1
This could be an issue with anti-virus software or gpg-agent... I got the same result with the docker build until I killed clamav gus@ns-l1:~$ ps aux | grep clamav clamav 1806 0.0 0.0 59124 14080 ?Ss 09:01 0:00 /usr/bin/freshclam -d --foreground=true gus11896 0.0 0.0 9212 2304 pts/0S+ 09:09 0:00 grep --color=auto clamav gus@ns-l1:~$ sudo kill -9 1806 [sudo] password for gus: gus@ns-l1:~$ ps aux | grep clamav gus11940 0.0 0.0 9080 2304 pts/0S+ 09:09 0:00 grep --color=auto clamav gus@ns-l1:~$ SOLR_DOWNLOAD_SERVER= https://dist.apache.org/repos/dist/dev/solr/solr-9.6.0-RC1-rev-f8e5a93c11267e13b7b43005a428bfb910ac6e57/solr && docker build $SOLR_DOWNLOAD_SERVER/9.6.0/docker/Dockerfile.official-full --build-arg SOLR_DOWNLOAD_SERVER=$SOLR_DOWNLOAD_SERVER -t solr-rc:9.6.0-1 && docker build $SOLR_DOWNLOAD_SERVER/9.6.0/docker/Dockerfile.official-slim --build-arg SOLR_DOWNLOAD_SERVER=$SOLR_DOWNLOAD_SERVER -t solr-rc:9.6.0-1-slim [+] Building 29.1s (10/10) FINISHED docker:default => CACHED [internal] load remote build context 0.2s => [internal] load metadata for docker.io/library/eclipse-temurin:17-jre-jammy 0.3s => CACHED [1/7] FROM docker.io/library/eclipse-temurin:17-jre-jammy@sha256:1b646daef966395c93995e73347d4c7c726c9ddba8695e984cd8dcf5d8b5b253 0.0s => [2/7] RUN set -ex; apt-get update; apt-get -y --no-install-recommends install wget gpg gnupg dirmngr; rm -rf /var/lib/apt/lists/*; export SOLR_BINARY="solr-9.6.0.tgz"; MAX_RED 20.7s => [3/7] RUN set -ex; groupadd -r --gid "8983" "solr"; useradd -r --uid "8983" --gid "8983" "solr" 0.4s => [4/7] RUN set -ex; (cd /opt; ln -s solr-*/ solr); rm -Rf /opt/solr/docs /opt/solr/docker/Dockerfile; 0.4s => [5/7] RUN set -ex; mkdir -p /opt/solr/server/solr/lib /docker-entrypoint-initdb.d; cp /opt/solr/bin/solr.in.sh /etc/default/ solr.in.sh; mv /opt/solr/bin/solr.in.sh /opt/solr/bin/so 0.5s => [6/7] RUN set -ex; apt-get update; apt-get -y --no-install-recommends install acl lsof procps wget netcat gosu tini jattach; rm -rf /var/lib/apt/lists/*; 5.3s => [7/7] WORKDIR /opt/solr 0.1s => exporting to image 0.9s => => exporting layers 0.9s => => writing image sha256:e426f6991bd325a5b01a31e90f3304ce45a787004c34fc6a90a2aa86c5193afc 0.0s => => naming to docker.io/library/solr-rc:9.6.0-1 0.0s [+] Building 19.3s (10/10) FINISHED docker:default => [internal] load remote build context 0.2s => [internal] load metadata for docker.io/library/eclipse-temurin:17-jre-jammy 0.2s => CACHED [1/7] FROM docker.io/library/eclipse-temurin:17-jre-jammy@sha256:1b646daef966395c93995e73347d4c7c726c9ddba8695e984cd8dcf5d8b5b253 0.0s => [2/7] RUN set -ex; apt-get update; apt-get -y --no-install-recommends install wget gpg gnupg dirmngr; rm -rf /var/lib/apt/lists/*; export SOLR_BINARY="solr-9.6.0-slim.tgz"; MA 11.3s => [3/7] RUN set -ex; groupadd -r --gid "8983" "solr"; useradd -r --uid "8983" --gid "8983" "solr" 0.5s => [4/7] RUN set -ex; (cd /opt; ln -s solr-*/ solr); rm -Rf /opt/solr/docs /opt/solr/docker/Dockerfile; 0.5s => [5/7] RUN set -ex; mkdir -p /opt/solr/server/solr/lib /docker-entrypoint-initdb.d; cp /opt/solr/bin/solr.in.sh /etc/default/ solr.in.sh; mv /opt/solr/bin/solr.in.sh /opt/solr/bin/so 0.5s => [6/7] RUN set -ex; apt-get update; apt-get -y --no-install-recommends install acl lsof procps wget netcat gosu tini jattach; rm -rf /var/lib/apt/lists/*; 5.6s => [7/7] WORKDIR /opt/solr 0.1s => exporting to image 0.5s => => exporting layers 0.4s => => writing image sha256:4933f3684276f1f7e239046e4978ab223df902f428b7af3a20300fca264a94c3 0.0s => => naming to docker.io/library/solr-rc:9.6.0-1-slim On Tue, Apr 23, 2024 at 7:11 AM Jan Høydahl wrote: > Hi, > > Smoketester succeeds: > SUCCESS! [0:45:28.467115] > > But I cannot get the docker build to pass: > It seems to fail during GPG validation of your key during image build: > > > gpg: key 140BC45803B03F7F: public key "Patrick Gustav Heck (CODE SIGNING > KEY) " imported > > gpg: can't connect to the agent: IPC connect call failed > > gpg: Total number pro
Re: Ideas for release notes for 9.6
For the second item, instead of "Solrj now offers async HTTP/2 Requests", it would be better to say "SolrJ now offers an improved async request API". We previously supported async but SOLR-14763 merely offers a better API and deprecates the existing one. I realize the JIRA title is not so accurate for what we actually did! On Mon, Apr 22, 2024 at 9:48 PM Gus Heck wrote: > > Initial release notes have been drafted here, please flesh out, refine, > copy edit as needed. > > https://cwiki.apache.org/confluence/display/SOLR/ReleaseNote9_6_0 > > -Gus > > -- > http://www.needhamsoftware.com (work) > https://a.co/d/b2sZLD9 (my fantasy fiction book) - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
Re: [VOTE] Release Solr 9.6.0 RC1
As a side note the clamav process appears to have been a red herring. For me the command succeeds ~40-50% of the time (after adding --no-cache) no idea why. On Tue, Apr 23, 2024 at 9:23 AM Gus Heck wrote: > This could be an issue with anti-virus software or gpg-agent... I got the > same result with the docker build until I killed clamav > > gus@ns-l1:~$ ps aux | grep clamav > clamav 1806 0.0 0.0 59124 14080 ?Ss 09:01 0:00 > /usr/bin/freshclam -d --foreground=true > gus11896 0.0 0.0 9212 2304 pts/0S+ 09:09 0:00 grep > --color=auto clamav > gus@ns-l1:~$ sudo kill -9 1806 > [sudo] password for gus: > gus@ns-l1:~$ ps aux | grep clamav > gus11940 0.0 0.0 9080 2304 pts/0S+ 09:09 0:00 grep > --color=auto clamav > gus@ns-l1:~$ SOLR_DOWNLOAD_SERVER= > https://dist.apache.org/repos/dist/dev/solr/solr-9.6.0-RC1-rev-f8e5a93c11267e13b7b43005a428bfb910ac6e57/solr > && docker build > $SOLR_DOWNLOAD_SERVER/9.6.0/docker/Dockerfile.official-full --build-arg > SOLR_DOWNLOAD_SERVER=$SOLR_DOWNLOAD_SERVER -t solr-rc:9.6.0-1 && > docker build $SOLR_DOWNLOAD_SERVER/9.6.0/docker/Dockerfile.official-slim > --build-arg SOLR_DOWNLOAD_SERVER=$SOLR_DOWNLOAD_SERVER -t > solr-rc:9.6.0-1-slim > [+] Building 29.1s (10/10) FINISHED > > docker:default > => CACHED [internal] load remote build context > > 0.2s > => [internal] load metadata for > docker.io/library/eclipse-temurin:17-jre-jammy > > 0.3s > => CACHED [1/7] FROM > docker.io/library/eclipse-temurin:17-jre-jammy@sha256:1b646daef966395c93995e73347d4c7c726c9ddba8695e984cd8dcf5d8b5b253 >0.0s > => [2/7] RUN set -ex; apt-get update; apt-get -y > --no-install-recommends install wget gpg gnupg dirmngr; rm -rf > /var/lib/apt/lists/*; export SOLR_BINARY="solr-9.6.0.tgz"; MAX_RED > 20.7s > => [3/7] RUN set -ex; groupadd -r --gid "8983" "solr"; useradd -r > --uid "8983" --gid "8983" "solr" > 0.4s > => [4/7] RUN set -ex; (cd /opt; ln -s solr-*/ solr); rm -Rf > /opt/solr/docs /opt/solr/docker/Dockerfile; >0.4s > => [5/7] RUN set -ex; mkdir -p /opt/solr/server/solr/lib > /docker-entrypoint-initdb.d; cp /opt/solr/bin/solr.in.sh /etc/default/ > solr.in.sh; mv /opt/solr/bin/solr.in.sh /opt/solr/bin/so 0.5s > => [6/7] RUN set -ex; apt-get update; apt-get -y > --no-install-recommends install acl lsof procps wget netcat gosu tini > jattach; rm -rf /var/lib/apt/lists/*; 5.3s > => [7/7] WORKDIR /opt/solr > > 0.1s > => exporting to image > > 0.9s > => => exporting layers > > 0.9s > => => writing image > sha256:e426f6991bd325a5b01a31e90f3304ce45a787004c34fc6a90a2aa86c5193afc > >0.0s > => => naming to docker.io/library/solr-rc:9.6.0-1 > >0.0s > [+] Building 19.3s (10/10) FINISHED > > docker:default > => [internal] load remote build context > > 0.2s > => [internal] load metadata for > docker.io/library/eclipse-temurin:17-jre-jammy > > 0.2s > => CACHED [1/7] FROM > docker.io/library/eclipse-temurin:17-jre-jammy@sha256:1b646daef966395c93995e73347d4c7c726c9ddba8695e984cd8dcf5d8b5b253 >0.0s > => [2/7] RUN set -ex; apt-get update; apt-get -y > --no-install-recommends install wget gpg gnupg dirmngr; rm -rf > /var/lib/apt/lists/*; export SOLR_BINARY="solr-9.6.0-slim.tgz"; MA > 11.3s > => [3/7] RUN set -ex; groupadd -r --gid "8983" "solr"; useradd -r > --uid "8983" --gid "8983" "solr" > 0.5s > => [4/7] RUN set -ex; (cd /opt; ln -s solr-*/ solr); rm -Rf > /opt/solr/docs /opt/solr/docker/Dockerfile; >0.5s > => [5/7] RUN set -ex; mkdir -p /opt/solr/server/solr/lib > /docker-entrypoint-initdb.d; cp /opt/solr/bin/solr.in.sh /etc/default/ > solr.in.sh; mv /opt/solr/bin/solr.in.sh /opt/solr/bin/so 0.5s > => [6/7] RUN set -ex; apt-get update; apt-get -y > --no-install-recommends install acl lsof procps wget netcat gosu tini > jattach; rm -rf /var/lib/apt/lists/*; 5.6s > => [7/7] WORKDIR /opt/solr > > 0.1s > => exporting to image > > 0.5s > => => exporting layers > > 0.4s > => => writing image > sha256:4933f3684276f1f7e239046e4978ab223df902f428b7af3a20300fca264a94c3 > >0.0s > => => naming to docker.io/library/solr-rc:9
Replicas-on-demand
I’m soliciting interest / feedback on something. Maybe some of you are familiar with transient-cores (SOLR-1028), an LRU cache SolrCore mechanism that allows you to have an almost unlimited number of cores on a node in name-only with a limited number that are actually loaded at any one time. It’s for use-cases where the working set is significantly smaller than the total possible addressable set. Transient-cores only works in standalone Solr; I tried to get it to work in SolrCloud but it proved difficult / buggy, especially with leader election entanglements. Furthermore, if we imagine tens of thousands of replicas on a node, actually maintaining that in SolrCloud / ZooKeeper is a ton of information and book-keeping / watching etc. I am imagining another approach where replicas are created and removed on-demand and thus the underlying core as well. And at a higher level of abstraction (at the request level) that can make more informed decisions than the “SolrCores”/transient-cores mechanism can. If a request comes in and we have 0 replicas and a shared file system, create one similar to autoAddReplicas[1]. If we have 1 replica, maybe we should asynchronously arrange for another to maintain good availability. If the core seems saturated with query activity, create another (to have more than 2 total). That might depend on /select vs /update and be replica-type specific. Meanwhile a node listener can remove replicas that have not been used recently, especially to limit the number of replicas per node. It can consider the number of *other* replicas for the same shard that exists, and leadership and replica type in its decision. Of course different users/apps might want to tune such settings differently, and it doesn’t imply a shared file system to be useful either. To support such a feature, I don’t think much is needed of Solr. The request “demand driven” aspect suggests a new plugin type within HttpSolrCall that resolves a request to a SolrCore, perhaps called a “RequestToCoreResolver”, or we make HttpSolrCall itself more extensible. For the 0-replica scenario, CloudSolrClient probably needs a little more tolerance to just get the request to Solr instead of prematurely failing. HttpShardHandler (for distributed-search) might similarly. There is no core-listener; it could be added, or we do polling, or we extend SolrCores as a collaborating plugin. One risk/concern is ensuring the core data is retained after the replica is removed. It’s not necessary to do that but if it’s removed, then it’s expensive & slow to create it again — a problem if there are no replicas to serve a request. I haven’t yet checked on the feasibility of keeping data lying around, and using it again when re-creating the replica. A significant motivation of mine with this proposal is to help SIP-20 “Zero Replicas” [2]. The biggest obstacle I see with it is that unused cores are empty until queried, yet still are in state ACTIVE the whole time (don’t even use the RECOVERY state). A hack in SearchHandler throws an exception and gets the data. This stuff doesn’t *need* to be in that branch (it’s not fundamental to Zero even if it’s fundamental to how we scale with Zero right now) but a replica-on-demand approach would obsolete that. [1]: Solr used to have an “autoAddReplicas” feature prior to Solr 9. In Solr 9 a substitute was developed — CollectionsRepairEventListener. Regardless, the idea is to create replicas automatically in response to nodes going away (or maybe other circumstances) in order to maintain a replicationFactor. There are many references to autoAddReplicas in CHANGES.txt; originally in SOLR-5656 it was intended for shared file system but was later expanded to be more general SOLR-10397. In the case of a shared file system, you can even reach 0 replicas and nonetheless create more later. [2]: SIP-20 https://cwiki.apache.org/confluence/x/8YokEQ and branch https://github.com/apache/solr/tree/jira/solr-17125-zero-replicas ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
