Eric
Thanks for your quick response.
So in the solrconfig.xml keep the out of the box setting of 15 seconds
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
and also have the <autoSoftCommit> setting set to something like 5 minutes
<autoSoftCommit>
<maxTime>300000</maxTime>
</autoSoftCommit>
Then in the existing SolrJ code just simply delete the line
up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
Is this what you recommended I try.
Thanks
Tyrone
On Tue, Jul 21, 2020 at 1:16 PM Erick Erickson <erickerick...@gmail.com>
wrote:
> What you’re seeing is the input from the client. You’ve passed true, true,
> which are waitFlush and waitSearcher
> which sets softCommit _for that call_ to false. It has nothing to do with
> the settings in your config file.
>
> bq. I am not passing the parameter to do a softCommit in the SolrJ
> command.
>
> I don’t think so. That’s a hard commit. This is a little tricky since the
> waitFlush and waitSearcher params don’t
> tell you that they are about hard commits. There used to only be hard
> commits, so...
>
> But…. these settings are highly suspicious. Here’s the long form:
>
>
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> It is risky to have your autocommit settings commented out. You risk
> transaction logs growing
> forever to no purpose whatsoever.
>
> Your call does a hard commit _and_ opens a new searcher for, apparently,
> every document.
>
> But your autoSoftCommit settings also open a new searcher without doing
> anything about flushing
> data to disk every second. Usually, this is far too often unless you have
> extremely stringent latency
> requirements, and in this case unless you’re only indexing once in a great
> while your caches
> are pretty useless.
>
> I strongly urge you to uncomment autocommit settings. Make the autoCommit
> interval something
> reasonable (15-60 seconds for instance) with openSearcher=false.
>
> Then lengthen your autoSoftCommit settings to as long as you can stand.
> The longer the interval,
> the less work you’ll do opening new searchers, which is a rather expensive
> operation. I like
> 5-10 minutes if possible, but your app may require shorter intervals.
>
> Then don’t send any commit settings in your SolrJ program at all.
>
> Best,
> Erick
>
> > On Jul 21, 2020, at 1:32 PM, Tyrone Tse <tyrone...@hotmail.com> wrote:
> >
> > I am using Solr 8.5 cloud, and in my collection I have edited the
> > solrconfig.xml file to use
> > <autoSoftCommit>
> > <maxTime>1000</maxTime>
> > </autoSoftCommit>
> >
> > and commented out the default <autoCommit> configuration
> >
> > <!--
> > <autoCommit>
> > <maxTime>15000</maxTime>
> > <openSearcher>false</openSearcher>
> > </autoCommit>
> > -->
> >
> > We are using SolrJ to post files to the Solr here is the snippet of Java
> > code that does it
> >
> > try(HttpSolrClient solrClient = solr.build()){
> > ContentStreamUpdateRequest up = new
> > ContentStreamUpdateRequest("/update/extract");
> > up.addFile(f, mimeType);
> > String tempId = f.getName() + (new Date()).toString();
> > up.setParam("literal.id", tempId);
> > up.setParam("literal.username", user);
> > up.setParam("literal.fileName", f.getName());
> > up.setParam("literal.filePath", path);
> > up.setParam("uprefix", "attr_");
> > up.setParam("fmap.content", "attr_content");
> > up.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
> > logger.info("PreRequest");
> > solrClient.request(up);
> > logger.info("PostRequest");
> > resultId = tempId;
> > } catch (IOException | SolrServerException |
> > HttpSolrClient.RemoteSolrException e) {
> > logger.error("Error connecting.committing to Solr", e);
> > }
> >
> > So I am not passing the parameter to do a softCommit in the SolrJ
> command.
> >
> > When I posted a file to my Solr core, when I look at the solr.log file I
> > see the following information
> >
> > 2020-07-21 16:38:54.719 INFO (qtp1546693040-302) [c:files s:shard1
> > r:core_node5 x:files_shard1_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory
> > [files_shard1_replica_n2] webapp=/solr path=/update
> >
> params={update.distrib=TOLEADER&update.chain=files-update-processor&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=
> > http://192.168.1.191:8983/solr/files_shard2_replica_n6/
> >
> > Does having <autoSoftCommit> set in the solrconfig.xml override REST
> Post
> > calls that have the parameter softCommit=false and force a softCommit
> when
> > the data is posted to Solr.
> >
> > Thanks in advance.
>
>
