Re: SolrCloud shows cluster still healthy even the node data directory is deleted
Hi Erick, I understand this is how the file handler works. But for the SolrCloud users, they didn't see the expected replica failover happens, then we can not say SolrCloud is totally HA enabled. Do we have plan to handle the HA for disk failures? Thanks. Regards, Radar From: Amy Bai Date: Wednesday, November 11, 2020 at 8:19 PM To: solr-user@lucene.apache.org Subject: Re: SolrCloud shows cluster still healthy even the node data directory is deleted Hi Erick, Thanks for your kindly reply. There are two things that confuse me: 1. index/search queries keep failing because one of the node data directory is gone, but the node is not marked as down. 2. The replicas on the failed node are not working, but the Index/search queries didn't failover to other healthy replicas. Regards, Amy From: Erick Erickson Sent: Monday, November 9, 2020 8:43 PM To: solr-user@lucene.apache.org Subject: Re: SolrCloud shows cluster still healthy even the node data directory is deleted Depends. *nix systems have delete-on-close semantics, that is as long as there’s a single file handle open, the file will be still be available to the process using it. Only when the last file handle is closed will the file actually be deleted. Solr (Lucene actually) has file handle open to every file in the index all the time. These files aren’t visible when you do a directory listing. So if you stop Solr, are the files gone? NOTE: When you start Solr again, if there are existing replicas that are healthy then the entire index should be copied from another replica…. Best, Erick > On Nov 9, 2020, at 3:30 AM, Amy Bai wrote: > > Hi community, > > I found that SolrCloud won't check the IO status if the SolrCloud process is > alive. > E.g. If I delete the SolrCloud data directory, there are no errors report, > and I can still log in to the SolrCloud Admin UI to create/query > collections. > Is this reasonable? > Can someone explain why SOLR handles it like this? > Thanks so much. > > > Regards, > Amy
Suggester using up memory
Hey, We have a problem on one of our installations with the suggestComponent. The index has about 16 million documents and contains a "Global" field which contains the data of multiple other fields. This "Global" field is used to build up the suggestions. A short time after starting Solr it is killed by oom_solr as it has run out of memory. We configured solr to use 8GB max for the heap, which is a lot more than what we use in other installations. Configuration of the suggestComponent: default Global FreeTextLookupFactory suggester_fuzzy_dir HighFrequencyDictionaryFactory TextField TextField false false Is there a problem in our configuration?How can we make sure the build up of the suggester does not run out of memory? How do we determine the actual memory requirements in this case? Thanks in advance! Nick
Atomic update wrongly deletes child documents
Hi, when I try to atomically update a single field of a parent/root document, all of its nested child documents disappear (Solr 8.6.3). I've tracked the problem down to the reconstruction of the original document in DistributedUpdateProcessor#getUpdatedDocument. Solr correctly finds existing nested documents, but skips them because our schema has a catch-all dynamic field to ignore unkown fields: multiValued="true" class="solr.StrField" /> (We use this to avoid errors about unknown fields from the indexing application, and I'd like to keep that.) However, this causes RealTimeGetComponent#toSolrInputDocument to ignore nested documents (source at [1]). It finds the "ignored" SchemaField as matching field for the nested document name (loaded from _nest_path_). I'm looking for a workaround now. Would it be possible to define some field for the nested documents in the schema? This would be quite a hack and I don't even know which field type to use. The reference guide does not recommend such a definition, with good reason [2]: > Even though child documents are provided as field values syntactically and with SolrJ, it’s a matter of syntax and it isn’t an actual field in the schema. Consequently, the field need not be defined in the schema and probably shouldn’t be as it would be confusing. There is no child document field type, at least not yet. I haven't found anything about this conflict with ignored fields in the documentation. Maybe it would make sense to treat ignored fields (non-stored,non-indexed,...) differently in RealTimeGetComponent? Or at least document this as caveat with some workaround? Any hints appreciated. Thanks, Andreas [1] https://github.com/apache/lucene-solr/blob/bf7db012ec31e6ef553b88abd2c2f75dc0298615/solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java#L740-L741 [2] https://lucene.apache.org/solr/guide/8_6/indexing-nested-documents.html#schema-configuration
SOLR 8.6.0 date Indexing Issues.
Hello Experts, I am having issues with indexing Date field in SOLR 8.6.0. I am indexing from MongoDB. In MongoDB the Format is as follows * "R_CREATION_DATE" : "12-Jul-18", "R_MODIFY_DATE" : "30-Apr-19", * In my Managed Schema I have the following entries. . I am getting an error in the Solr log. * org.apache.solr.common.SolrException: ERROR: [doc=mt_100] Error adding field 'R_MODIFY_DATE'='15-Jul-19' msg=Couldn't parse date because: Improperly formatted datetime: 15-Jul-19* Please let me know how to handle this usecase with Date format "12-JUL-18". what changes should I do to make it work ? Thanks Fiz N.
Re: SOLR 8.6.0 date Indexing Issues.
Your should format the date according to the ISO Standard: https://lucene.apache.org/solr/guide/6_6/working-with-dates.html Eg. 2018-07-12T00:00:00Z You can either transform the date that you have in Solr or in your client pushing the doc to Solr. All major programming language have date utilities that allow you do to this transformation easily. > Am 20.11.2020 um 21:50 schrieb Fiz N : > > Hello Experts, > > I am having issues with indexing Date field in SOLR 8.6.0. I am indexing > from MongoDB. In MongoDB the Format is as follows > > > * "R_CREATION_DATE" : "12-Jul-18", "R_MODIFY_DATE" : "30-Apr-19", * > > In my Managed Schema I have the following entries. > > > > > . > > I am getting an error in the Solr log. > > * org.apache.solr.common.SolrException: ERROR: [doc=mt_100] Error adding > field 'R_MODIFY_DATE'='15-Jul-19' msg=Couldn't parse date because: > Improperly formatted datetime: 15-Jul-19* > > Please let me know how to handle this usecase with Date format > "12-JUL-18". what changes should I do to make it work ? > > Thanks > Fiz N.
solr 8.6.3 and noggit
Hi, got this error using streaming with solrj 8.6.3 . does it use noggit-0.8. It was not mentioned in dependencies https://github.com/apache/lucene-solr/blob/branch_8_6/solr/solrj/ivy.xml Caused by: java.lang.NoSuchMethodError: 'java.lang.Object org.noggit.ObjectBuilder.getValStrict()' at org.apache.solr.common.util.Utils.fromJSON(Utils.java:284) ~[solr-solrj-8.6.3.jar:8.6.3 e001c2221812a0ba9e9378855040ce72f93eced4 - jasongerlowski - 2020-10-03 18:12:06]
Re: solr 8.6.3 and noggit
Noggit code was forked into Solr, see SOLR-13427 https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.3/solr/solrj/src/java/org/noggit/ObjectBuilder.java It looks like that particular method was added in 8.4 via https://issues.apache.org/jira/browse/SOLR-13824 Is it possible you're using an older SolrJ against a newer Solr server (or vice versa). Mike On Fri, Nov 20, 2020 at 2:25 PM Susmit Shukla wrote: > Hi, > got this error using streaming with solrj 8.6.3 . does it use noggit-0.8. > It was not mentioned in dependencies > https://github.com/apache/lucene-solr/blob/branch_8_6/solr/solrj/ivy.xml > > Caused by: java.lang.NoSuchMethodError: 'java.lang.Object > org.noggit.ObjectBuilder.getValStrict()' > > at org.apache.solr.common.util.Utils.fromJSON(Utils.java:284) > ~[solr-solrj-8.6.3.jar:8.6.3 e001c2221812a0ba9e9378855040ce72f93eced4 - > jasongerlowski - 2020-10-03 18:12:06] >
Re: solr 8.6.3 and noggit
Thanks Mike That explains it, just removing the noggit-0.6 jar should fix it. This error depended on classloading order and didn't show up on mac but was a problem on linux. On Fri, Nov 20, 2020 at 2:54 PM Mike Drob wrote: > Noggit code was forked into Solr, see SOLR-13427 > > https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.3/solr/solrj/src/java/org/noggit/ObjectBuilder.java > > It looks like that particular method was added in 8.4 via > https://issues.apache.org/jira/browse/SOLR-13824 > > Is it possible you're using an older SolrJ against a newer Solr server (or > vice versa). > > Mike > > On Fri, Nov 20, 2020 at 2:25 PM Susmit Shukla > wrote: > > > Hi, > > got this error using streaming with solrj 8.6.3 . does it use noggit-0.8. > > It was not mentioned in dependencies > > https://github.com/apache/lucene-solr/blob/branch_8_6/solr/solrj/ivy.xml > > > > Caused by: java.lang.NoSuchMethodError: 'java.lang.Object > > org.noggit.ObjectBuilder.getValStrict()' > > > > at org.apache.solr.common.util.Utils.fromJSON(Utils.java:284) > > ~[solr-solrj-8.6.3.jar:8.6.3 e001c2221812a0ba9e9378855040ce72f93eced4 - > > jasongerlowski - 2020-10-03 18:12:06] > > >
