Re: Integrating Solr with my existing java web application

[Jörn Franke]

I recommend to integrate log4j2 into the app instead of using println. Then you 
will see all the log statements including the one of Solr in a log file that 
will indicate you the issue.

> Am 01.11.2019 um 07:46 schrieb Khare, Kushal (MIND) 
> :
> 
> Hello mates !
> Hope you people are doing good !
> 
> Well, I am trying to integrate the SolrJ code for indexing and querying the 
> documents through Solr with my java web app. I am facing a very wired issue, 
> that when I run my method for Solr as java app (independently using main() 
> function) it works fine- extracts text , searches. But, as soon as I give the 
> hit through my web app from search page, it hits, function is being called, 
> but it does not extracts any text from docs, and is unable to search.
> 
> I verified all the jars, etc. but could not figure out anything. Following is 
> the code that I am using :
> 
> 
> package com.mind.qdms.utility;
> 
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.IOException;
> import java.io.InputStream;
> import java.util.ArrayList;
> import java.util.Collection;
> import java.util.HashMap;
> import java.util.List;
> import java.util.Map;
> import org.apache.solr.client.solrj.SolrServerException;
> import org.apache.solr.client.solrj.impl.HttpSolrClient;
> import org.apache.solr.client.solrj.response.QueryResponse;
> import org.apache.solr.client.solrj.response.UpdateResponse;
> import org.apache.solr.common.SolrDocument;
> import org.apache.solr.common.SolrDocumentList;
> import org.apache.solr.common.SolrInputDocument;
> import org.apache.solr.common.params.MapSolrParams;
> import org.apache.tika.metadata.Metadata;
> import org.apache.tika.parser.AutoDetectParser;
> import org.apache.tika.parser.ParseContext;
> import org.apache.tika.sax.BodyContentHandler;
> import org.xml.sax.ContentHandler;
> 
> 
> public class QdmsSolrUtilityMethods {
> 
> 
> 
> public static void main(String[] args) throws IOException,
> SolrServerException { QdmsSolrUtilityMethods.getDocsList("kushal"); }
> 
> 
>   public static List getDocsList(String keyword) throws 
> IOException, SolrServerException {
>  System.out.println("in util");
>  HttpSolrClient client = new 
> HttpSolrClient.Builder("http://localhost:8983/solr/tika";).build();
>  AutoDetectParser autoParser = new AutoDetectParser();
>  indexTikaDocuments(new File("D:\\docs"), client, autoParser);
>  List resultDocList = queryDocuments(client, keyword);
>  return resultDocList;
>   }
> 
>   public static void indexTikaDocuments(File root,HttpSolrClient
> client,AutoDetectParser autoParser) throws IOException, 
> SolrServerException {
> int totalTika = 0;
> 
> @SuppressWarnings("rawtypes") Collection docList = new ArrayList();
> 
> for (File file : root.listFiles()){
>if (file.isDirectory()) {
>   indexTikaDocuments(file,client,autoParser);
>   continue;
>}
>ContentHandler textHandler = new BodyContentHandler(-1);
>Metadata metadata = new Metadata();
>ParseContext context = new ParseContext();
>InputStream input = new
>FileInputStream(file);
>try {
>autoParser.parse(input, textHandler, metadata, context);
>}catch (Exception e) {
>System.out.println(String.format("File %s failed", 
> file.getCanonicalPath()));
>e.printStackTrace();
>continue;
>}
>SolrInputDocument doc = new SolrInputDocument();
>doc.addField("id", file.getCanonicalPath());
>doc.addField("_text_", textHandler.toString());
>docList.add(doc);
>System.out.println(textHandler.toString());
>System.out.println( file.getCanonicalPath()); ++totalTika;
>// Completely arbitrary, just batch up more than one document 
> for throughput!
>if(docList.size() >= 1000) {
>   // Commit within 5 minutes.
>   UpdateResponse resp = client.add(docList, 30);
>   if (resp.getStatus() != 0) {
> System.out.println("Some horrible error has 
> occurred, status is: " +
> resp.getStatus());
> }
>   docList.clear();
>   }
>}if(docList.size() > 0) {
>   client.add(docList, 30);
> } client.commit();
>System.out.println("indexed " + totalTika + " documents");
>}
> 
>   public static List queryDocuments(HttpSolrClient client1, 
> String queryTerm) throws SolrServerException, IOException {
>  List resultList = n

Re: Query on stemming

[Jörn Franke]

How did you define the field type? Probably you have syntax errors there. I 
recommend to use the schema rest api instead of schema xml as it will give you 
better feedback on what is wrong and it allows you also better versioning of 
the schema in a source code repository.

https://lucene.apache.org/solr/guide/8_2/schema-api.html


> Am 01.11.2019 um 06:41 schrieb Shubham Goswami :
> 
> Hello Community
> 
> I am using a filter class EnglishPorterFilterFactory for stemming filter
> but because of usage of this class, my solr is not able reload the schema.
> 
> Can somebody please let me know what exactly this class does and how can i
> implement stemming ?
> Any help will be appreciated. Thanks in advance.
> 
> -- 
> Kind Regards,
> Shubham Goswami
> Enterprise Software Engineer
> mobile: +91 7803886288
> email: *shubham.gosw...@hotwax.co*
> *www.hotwax.co *

Re: Query on stemming

[Shubham Goswami]

Hi Jorn

Thanks for your response.
Following is my field type definition and i am getting the
ClassNotFoundException while indexing after making the changes.

  

  
  
  
  


  
  
  
  
 
  

On Fri, Nov 1, 2019 at 2:10 PM Jörn Franke  wrote:

> How did you define the field type? Probably you have syntax errors there.
> I recommend to use the schema rest api instead of schema xml as it will
> give you better feedback on what is wrong and it allows you also better
> versioning of the schema in a source code repository.
>
> https://lucene.apache.org/solr/guide/8_2/schema-api.html
>
>
> > Am 01.11.2019 um 06:41 schrieb Shubham Goswami <
> shubham.gosw...@hotwax.co>:
> >
> > Hello Community
> >
> > I am using a filter class EnglishPorterFilterFactory for stemming filter
> > but because of usage of this class, my solr is not able reload the
> schema.
> >
> > Can somebody please let me know what exactly this class does and how can
> i
> > implement stemming ?
> > Any help will be appreciated. Thanks in advance.
> >
> > --
> > Kind Regards,
> > Shubham Goswami
> > Enterprise Software Engineer
> > mobile: +91 7803886288
> > email: *shubham.gosw...@hotwax.co*
> > *www.hotwax.co *
>


-- 
Kind Regards,
Shubham Goswami
Enterprise Software Engineer
mobile: +91 7803886288
email: *shubham.gosw...@hotwax.co*
*www.hotwax.co *

Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

[Shalin Shekhar Mangar]

Perhaps this bug in the kernel?
https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64

Can you check if you are running an affected kernel?

On Fri, Nov 1, 2019 at 7:09 AM Daniel de Oliveira Mantovani <
daniel.oliveira.mantov...@gmail.com> wrote:

> I have a Solr 7.6.0 Cloud with 4 nodes and while indexing Solr hangs.
>
> $ java -version
> java version "1.8.0_231"
> Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode
>
> $ sudo strace -f -c -p 5972
> % time seconds  usecs/call callserrors syscall
> -- --- --- - - 
>  71.02   33.329124 662 50335 13317 futex
>  21.43   10.055086  3351703028 restart_syscall
>   7.493.512846   14760   238   epoll_wait
>   0.070.030526254412   poll
>   0.000.001571   5   322   write
>   0.000.000439   4   117   times
>   0.000.000355   2   183 4 read
>   0.000.000236   372   mprotect
>   0.000.000161   1   118   epoll_ctl
>   0.000.000126   0   352   stat
>   0.000.89  11 8   getdents
>   0.000.77  10 8   sendto
>   0.000.69  69 1   statfs
>   0.000.43  22 2   writev
>   0.000.29   4 7   openat
>   0.000.21   4 6   fstat
>   0.000.18   122   lstat
>   0.000.18   9 2   dup
>   0.000.10   1 9   close
>   0.000.10   3 4   fcntl
>   0.000.05   3 2   access
>   0.000.02   2 1   mmap
>   0.000.02   026   sched_yield
>   0.000.01   0 8   recvfrom
>   0.000.01   1 2   readlink
> -- --- --- - - 
> 100.00   46.930865 51887 13349 total
>
> $ sudo strace -f -p 5972
> strace: Process 5972 attached with 86 threads
> [pid  7524] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  7523] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  7511] epoll_wait(142,  
> [pid  7471] epoll_wait(160,  
> [pid  7266] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  7267] epoll_wait(151,  
> [pid  7264] futex(0x7fcbd8f3eb98, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  7265] epoll_wait(157,  
> [pid  6311] futex(0x7fcabc8c42f8, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6310] futex(0x7fcab843c998, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6281] epoll_wait(154,  
> [pid  6280] futex(0x7fca68001d7c, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6278] futex(0x7fca74001d78, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6277] accept(136,  
> [pid  6272] futex(0x7fca7c001d7c, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6271] epoll_wait(208,  
> [pid  6270] epoll_wait(205,  
> [pid  6262] epoll_wait(199,  
> [pid  6261] epoll_wait(196,  
> [pid  6258] epoll_wait(193,  
> [pid  6269] epoll_wait(202,  
> [pid  6257] epoll_wait(190,  
> [pid  6256] epoll_wait(187,  
> [pid  6255] epoll_wait(184,  
> [pid  6253] epoll_wait(178,  
> [pid  6250] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  6252] epoll_wait(175,  
> [pid  6251] epoll_wait(172,  
> [pid  6254] epoll_wait(181,  
> [pid  6246] futex(0x7fcbd8f3f378, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6238] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  6237] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  6235] futex(0x7fcbd83d97b8, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6234] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  6233] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  6232] futex(0x7fcad8002608, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6231] futex(0x7fcbd89425ac, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6049] futex(0x7fcaf4002388, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6230] epoll_wait(134,  
> [pid  6048] futex(0x7fcbd8db9ac8, FUTEX_WAIT_PRIVATE, 0, NULL  ...>
> [pid  6047] epoll_wait(130,  
> [pid  6046] restart_syscall(<... resuming interrupted futex ...>
> 
> [pid  6045] restart_syscall(<... resuming interrupted futex ...>
> 
>
> JVM args:
>
>
>-
> -DSTOP.KEY=solrrocks-DSTOP.PORT=7983-Djetty.home=/opt/solr/server-Djetty.port=8983-Dlog4j.configurationFile=file:/opt/solr/server/resources/log4j2.xml-Dsolr.data.home=-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf-Dsolr.install.dir=/opt/solr-Dsolr.jetty.https.port=8983-Dsolr.log.dir=/opt/solr/example/cloud/node1/solr/../logs-Dsolr.log.muteconsole-Dsolr

Re: Query on stemming

[Jörn Franke]

https://lucene.apache.org/solr/guide/8_2/filter-descriptions.html

It looks to me that you write the name of the class wrongly 

> Am 01.11.2019 um 11:12 schrieb Shubham Goswami :
> 
> Hi Jorn
> 
> Thanks for your response.
> Following is my field type definition and i am getting the
> ClassNotFoundException while indexing after making the changes.
> 
>   positionIncrementGap="100" multiValued="true">
>
>  
>   ignoreCase="true"/>
>  
>  
>
>
>  
>   ignoreCase="true"/>
>   ignoreCase="true" synonyms="synonyms.txt"/>
>  
> 
>  
> 
>> On Fri, Nov 1, 2019 at 2:10 PM Jörn Franke  wrote:
>> 
>> How did you define the field type? Probably you have syntax errors there.
>> I recommend to use the schema rest api instead of schema xml as it will
>> give you better feedback on what is wrong and it allows you also better
>> versioning of the schema in a source code repository.
>> 
>> https://lucene.apache.org/solr/guide/8_2/schema-api.html
>> 
>> 
>>> Am 01.11.2019 um 06:41 schrieb Shubham Goswami <
>> shubham.gosw...@hotwax.co>:
>>> 
>>> Hello Community
>>> 
>>> I am using a filter class EnglishPorterFilterFactory for stemming filter
>>> but because of usage of this class, my solr is not able reload the
>> schema.
>>> 
>>> Can somebody please let me know what exactly this class does and how can
>> i
>>> implement stemming ?
>>> Any help will be appreciated. Thanks in advance.
>>> 
>>> --
>>> Kind Regards,
>>> Shubham Goswami
>>> Enterprise Software Engineer
>>> mobile: +91 7803886288
>>> email: *shubham.gosw...@hotwax.co*
>>> *www.hotwax.co *
>> 
> 
> 
> -- 
> Kind Regards,
> Shubham Goswami
> Enterprise Software Engineer
> mobile: +91 7803886288
> email: *shubham.gosw...@hotwax.co*
> *www.hotwax.co *

Re: Query on stemming

[Paras Lehana]

Hi Jorn,

It looks to me that you write the name of the class wrongly.


*EnglishPorterFilterFactory* did exist in older Solr versions
.
Since it was deprecated long ago, I think that's the reason it's not
mentioned in Filters Descriptions
. See,
even Ref Guide 7.7 mentions
 it in one of its
examples (maybe because the material was just copied).

On Fri, 1 Nov 2019 at 17:06, Jörn Franke  wrote:

> https://lucene.apache.org/solr/guide/8_2/filter-descriptions.html
>
> It looks to me that you write the name of the class wrongly
>
> > Am 01.11.2019 um 11:12 schrieb Shubham Goswami <
> shubham.gosw...@hotwax.co>:
> >
> > Hi Jorn
> >
> > Thanks for your response.
> > Following is my field type definition and i am getting the
> > ClassNotFoundException while indexing after making the changes.
> > 
> >   > positionIncrementGap="100" multiValued="true">
> >
> >  
> >   > ignoreCase="true"/>
> >  
> >  
> >
> >
> >  
> >   > ignoreCase="true"/>
> >   > ignoreCase="true" synonyms="synonyms.txt"/>
> >  
> > 
> >  
> >
> >> On Fri, Nov 1, 2019 at 2:10 PM Jörn Franke 
> wrote:
> >>
> >> How did you define the field type? Probably you have syntax errors
> there.
> >> I recommend to use the schema rest api instead of schema xml as it will
> >> give you better feedback on what is wrong and it allows you also better
> >> versioning of the schema in a source code repository.
> >>
> >> https://lucene.apache.org/solr/guide/8_2/schema-api.html
> >>
> >>
> >>> Am 01.11.2019 um 06:41 schrieb Shubham Goswami <
> >> shubham.gosw...@hotwax.co>:
> >>>
> >>> Hello Community
> >>>
> >>> I am using a filter class EnglishPorterFilterFactory for stemming
> filter
> >>> but because of usage of this class, my solr is not able reload the
> >> schema.
> >>>
> >>> Can somebody please let me know what exactly this class does and how
> can
> >> i
> >>> implement stemming ?
> >>> Any help will be appreciated. Thanks in advance.
> >>>
> >>> --
> >>> Kind Regards,
> >>> Shubham Goswami
> >>> Enterprise Software Engineer
> >>> mobile: +91 7803886288
> >>> email: *shubham.gosw...@hotwax.co*
> >>> *www.hotwax.co *
> >>
> >
> >
> > --
> > Kind Regards,
> > Shubham Goswami
> > Enterprise Software Engineer
> > mobile: +91 7803886288
> > email: *shubham.gosw...@hotwax.co*
> > *www.hotwax.co *
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

-- 
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.

Re: Query on stemming

[Jörn Franke]

Ok yes but I assumed that he is referring to a newer version as no version was 
mentioned 

> Am 01.11.2019 um 12:43 schrieb Paras Lehana :
> 
> Hi Jorn,
> 
> It looks to me that you write the name of the class wrongly.
> 
> 
> *EnglishPorterFilterFactory* did exist in older Solr versions
> .
> Since it was deprecated long ago, I think that's the reason it's not
> mentioned in Filters Descriptions
> . See,
> even Ref Guide 7.7 mentions
>  it in one of its
> examples (maybe because the material was just copied).
> 
>> On Fri, 1 Nov 2019 at 17:06, Jörn Franke  wrote:
>> 
>> https://lucene.apache.org/solr/guide/8_2/filter-descriptions.html
>> 
>> It looks to me that you write the name of the class wrongly
>> 
>>> Am 01.11.2019 um 11:12 schrieb Shubham Goswami <
>> shubham.gosw...@hotwax.co>:
>>> 
>>> Hi Jorn
>>> 
>>> Thanks for your response.
>>> Following is my field type definition and i am getting the
>>> ClassNotFoundException while indexing after making the changes.
>>> 
>>> >> positionIncrementGap="100" multiValued="true">
>>>   
>>> 
>>> >> ignoreCase="true"/>
>>> 
>>> 
>>>   
>>>   
>>> 
>>> >> ignoreCase="true"/>
>>> >> ignoreCase="true" synonyms="synonyms.txt"/>
>>> 
>>>
>>> 
>>> 
 On Fri, Nov 1, 2019 at 2:10 PM Jörn Franke 
>> wrote:
 
 How did you define the field type? Probably you have syntax errors
>> there.
 I recommend to use the schema rest api instead of schema xml as it will
 give you better feedback on what is wrong and it allows you also better
 versioning of the schema in a source code repository.
 
 https://lucene.apache.org/solr/guide/8_2/schema-api.html
 
 
> Am 01.11.2019 um 06:41 schrieb Shubham Goswami <
 shubham.gosw...@hotwax.co>:
> 
> Hello Community
> 
> I am using a filter class EnglishPorterFilterFactory for stemming
>> filter
> but because of usage of this class, my solr is not able reload the
 schema.
> 
> Can somebody please let me know what exactly this class does and how
>> can
 i
> implement stemming ?
> Any help will be appreciated. Thanks in advance.
> 
> --
> Kind Regards,
> Shubham Goswami
> Enterprise Software Engineer
> mobile: +91 7803886288
> email: *shubham.gosw...@hotwax.co*
> *www.hotwax.co *
 
>>> 
>>> 
>>> --
>>> Kind Regards,
>>> Shubham Goswami
>>> Enterprise Software Engineer
>>> mobile: +91 7803886288
>>> email: *shubham.gosw...@hotwax.co*
>>> *www.hotwax.co *
>> 
> 
> 
> -- 
> -- 
> Regards,
> 
> *Paras Lehana* [65871]
> Development Engineer, Auto-Suggest,
> IndiaMART Intermesh Ltd.
> 
> 8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
> Noida, UP, IN - 201303
> 
> Mob.: +91-9560911996
> Work: 01203916600 | Extn:  *8173*
> 
> -- 
> IMPORTANT: 
> NEVER share your IndiaMART OTP/ Password with anyone.

Re: Query on stemming

[Paras Lehana]

Hey Jorn,

Ok yes but I assumed that he is referring to a newer version as no version
> was mentioned.


You are right, this is probably the case! Shubham, have you just upgraded
or is it the first time you are using the filter? Switch to Snowball and
everything will be sorted.

On Fri, 1 Nov 2019 at 17:27, Jörn Franke  wrote:

> Ok yes but I assumed that he is referring to a newer version as no version
> was mentioned
>
> > Am 01.11.2019 um 12:43 schrieb Paras Lehana  >:
> >
> > Hi Jorn,
> >
> > It looks to me that you write the name of the class wrongly.
> >
> >
> > *EnglishPorterFilterFactory* did exist in older Solr versions
> > <
> https://lucene.apache.org/solr/3_6_0/org/apache/solr/analysis/EnglishPorterFilterFactory.html
> >.
> > Since it was deprecated long ago, I think that's the reason it's not
> > mentioned in Filters Descriptions
> > .
> See,
> > even Ref Guide 7.7 mentions
> >  it in one of
> its
> > examples (maybe because the material was just copied).
> >
> >> On Fri, 1 Nov 2019 at 17:06, Jörn Franke  wrote:
> >>
> >> https://lucene.apache.org/solr/guide/8_2/filter-descriptions.html
> >>
> >> It looks to me that you write the name of the class wrongly
> >>
> >>> Am 01.11.2019 um 11:12 schrieb Shubham Goswami <
> >> shubham.gosw...@hotwax.co>:
> >>>
> >>> Hi Jorn
> >>>
> >>> Thanks for your response.
> >>> Following is my field type definition and i am getting the
> >>> ClassNotFoundException while indexing after making the changes.
> >>> 
> >>>  >>> positionIncrementGap="100" multiValued="true">
> >>>   
> >>> 
> >>>  >>> ignoreCase="true"/>
> >>> 
> >>> 
> >>>   
> >>>   
> >>> 
> >>>  >>> ignoreCase="true"/>
> >>>  >>> ignoreCase="true" synonyms="synonyms.txt"/>
> >>> 
> >>>
> >>> 
> >>>
>  On Fri, Nov 1, 2019 at 2:10 PM Jörn Franke 
> >> wrote:
> 
>  How did you define the field type? Probably you have syntax errors
> >> there.
>  I recommend to use the schema rest api instead of schema xml as it
> will
>  give you better feedback on what is wrong and it allows you also
> better
>  versioning of the schema in a source code repository.
> 
>  https://lucene.apache.org/solr/guide/8_2/schema-api.html
> 
> 
> > Am 01.11.2019 um 06:41 schrieb Shubham Goswami <
>  shubham.gosw...@hotwax.co>:
> >
> > Hello Community
> >
> > I am using a filter class EnglishPorterFilterFactory for stemming
> >> filter
> > but because of usage of this class, my solr is not able reload the
>  schema.
> >
> > Can somebody please let me know what exactly this class does and how
> >> can
>  i
> > implement stemming ?
> > Any help will be appreciated. Thanks in advance.
> >
> > --
> > Kind Regards,
> > Shubham Goswami
> > Enterprise Software Engineer
> > mobile: +91 7803886288
> > email: *shubham.gosw...@hotwax.co*
> > *www.hotwax.co *
> 
> >>>
> >>>
> >>> --
> >>> Kind Regards,
> >>> Shubham Goswami
> >>> Enterprise Software Engineer
> >>> mobile: +91 7803886288
> >>> email: *shubham.gosw...@hotwax.co*
> >>> *www.hotwax.co *
> >>
> >
> >
> > --
> > --
> > Regards,
> >
> > *Paras Lehana* [65871]
> > Development Engineer, Auto-Suggest,
> > IndiaMART Intermesh Ltd.
> >
> > 8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
> > Noida, UP, IN - 201303
> >
> > Mob.: +91-9560911996
> > Work: 01203916600 | Extn:  *8173*
> >
> > --
> > IMPORTANT:
> > NEVER share your IndiaMART OTP/ Password with anyone.
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

-- 
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.

Re: Solr 7.6.0 High CPU Usage Deadlock (Generating 70% of futex system call)

[Daniel de Oliveira Mantovani]

Hello Shalin,

It's not an OS bug,
Linux samba 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
x86_64 x86_64 x86_64 GNU/Linux

I'm sending the full thread dump log:
2019-11-01 11:05:13
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode):

"Attach Listener" #259 daemon prio=9 os_prio=0 tid=0x7fcb78001000
nid=0x2659 waiting on condition [0x]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
- None

"qtp1571967156-258" #258 prio=5 os_prio=0 tid=0x7fcabc321000 nid=0x25c7
waiting on condition [0x7fcb356ca000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0005a626cdd8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-257" #257 prio=5 os_prio=0 tid=0x7fcb24001000 nid=0x25bc
waiting on condition [0x7fcb35b95000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0004f00242c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-256" #256 prio=5 os_prio=0 tid=0x7fca88003000 nid=0x2594
runnable [0x7fcb37441000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0004f0a34218> (a sun.nio.ch.Util$3)
- locked <0x0004f0a34200> (a java.util.Collections$UnmodifiableSet)
- locked <0x0004f0931118> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

"qtp1571967156-254" #254 prio=5 os_prio=0 tid=0x7fca88005800 nid=0x2570
runnable [0x7fcb3580f000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0004f0a2fdd8> (a sun.nio.ch.Util$3)
- locked <0x0004f0a2fdc0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0004f0243070> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at
org.eclipse.

Re: Query on stemming

[Paras Lehana]

Hi Shubham,

 I am getting the ClassNotFoundException while indexing after making the
> changes.



   - Please describe the error properly with full details. We need to see
   what is exactly causing ClassNotFoundException in the trace.
   - This seems to be a Java syntax error due to
   *EnglishPorterFilterFactory**.java* being not available. Please confirm
   if it's available in analysis directory.
   - Also, what Solr version are you using? *EnglishPorterFilterFactory* is
   already *deprecated* and I suggest you to use
   *SnowballPorterFilterFactory* with *language="English" *instead. You
   need to add the stemming in query time analysis chain too, otherwise, query
   as "bags" will not match with "bag" (indexed).


Hope this helps.

On Fri, 1 Nov 2019 at 15:42, Shubham Goswami 
wrote:

> Hi Jorn
>
> Thanks for your response.
> Following is my field type definition and i am getting the
> ClassNotFoundException while indexing after making the changes.
> 
>positionIncrementGap="100" multiValued="true">
> 
>   
>ignoreCase="true"/>
>   
>   
> 
> 
>   
>ignoreCase="true"/>
>ignoreCase="true" synonyms="synonyms.txt"/>
>   
>  
>   
>
> On Fri, Nov 1, 2019 at 2:10 PM Jörn Franke  wrote:
>
> > How did you define the field type? Probably you have syntax errors there.
> > I recommend to use the schema rest api instead of schema xml as it will
> > give you better feedback on what is wrong and it allows you also better
> > versioning of the schema in a source code repository.
> >
> > https://lucene.apache.org/solr/guide/8_2/schema-api.html
> >
> >
> > > Am 01.11.2019 um 06:41 schrieb Shubham Goswami <
> > shubham.gosw...@hotwax.co>:
> > >
> > > Hello Community
> > >
> > > I am using a filter class EnglishPorterFilterFactory for stemming
> filter
> > > but because of usage of this class, my solr is not able reload the
> > schema.
> > >
> > > Can somebody please let me know what exactly this class does and how
> can
> > i
> > > implement stemming ?
> > > Any help will be appreciated. Thanks in advance.
> > >
> > > --
> > > Kind Regards,
> > > Shubham Goswami
> > > Enterprise Software Engineer
> > > mobile: +91 7803886288
> > > email: *shubham.gosw...@hotwax.co*
> > > *www.hotwax.co *
> >
>
>
> --
> Kind Regards,
> Shubham Goswami
> Enterprise Software Engineer
> mobile: +91 7803886288
> email: *shubham.gosw...@hotwax.co*
> *www.hotwax.co *
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Development Engineer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

-- 
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.

Solr 8.2.0 - Unable to write response

[Joe Obernberger]

Hi All - getting this error from only one server in a 45 node cluster 
when calling COLSTATUS.  Any ideas?



2019-11-01 13:17:32.556 INFO  (qtp694316372-44709) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2} 
status=0 QTime=94734
2019-11-01 13:17:32.567 INFO  (qtp694316372-44688) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2} 
status=0 QTime=815338
2019-11-01 13:17:32.570 INFO  (qtp694316372-44688) [   ] 
o.a.s.s.HttpSolrCall Unable to write response, client closed connection 
or we are shutting down => org.eclipse.jetty.io.EofException: Closed

    at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
org.eclipse.jetty.io.EofException: Closed
    at 
org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.apache.solr.servlet.ServletOutputStreamWrapper.write(ServletOutputStreamWrapper.java:134) 
~[?:?]
    at 
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216) 
~[?:?]
    at 
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209) 
~[?:?]
    at 
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:170) 
~[?:?]
    at 
org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:60) 
~[?:?]
    at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49) 
~[?:?]
    at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:873) 
~[?:?]
    at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:793) 
~[?:?]
    at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:546) ~[?:?]
    at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423) 
~[?:?]
    at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350) 
~[?:?]
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) 
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) 
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) 
~[jetty-security-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) 
~[jetty-servlet-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
~[jetty-server-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335) 
~[jetty-rewrite-9.4.19.v20190610.jar:9.4.19.v20190610]
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.ha

Re: Security Vulnerability Consultation

[Erik Hatcher]

Hi -

There are many "vulnerabilities" that can be enabled when one has 
administrative access to Solr, with this being one example.   The setting 
mentioned defaults to false, and requires admin access to enable.

The warning from the Solr Reference Guide is worth repeating here:

>> No Solr API, including the Admin UI, is designed to be exposed to 
>> non-trusted parties. 

Turning on authentication is the first step I'd recommend.

Erik


> On Oct 31, 2019, at 11:45 PM, Huawei PSIRT  wrote:
> 
> Dear,
> 
> 
> 
>This is Huawei PSIRT. We have learned that a security researcher
>  released an
> Apache Solr RCE suspected vulnerability on October 31, 2019.
> 
>The links are as follow:
> https://meterpreter.org/unpatch-apache-solr-remote-command-execution-vulnera
> bility-alert/
> 
> https://gist.github.com/s00py/a1ba36a3689fa13759ff910e179fc133
> 
> 
> 
> We want to confirm if the issue exists. If it exists, when will the
> patches be released ?
> 
> Looking forward to your reply. Thank you.
> 
> 
> 
> Best Regards,
> 
> Huawei PSIRT
> 

Re: Solr 8.2.0 - Unable to write response

[Shawn Heisey]

On 11/1/2019 7:20 AM, Joe Obernberger wrote:
Hi All - getting this error from only one server in a 45 node cluster 
when calling COLSTATUS.  Any ideas?




2019-11-01 13:17:32.556 INFO  (qtp694316372-44709) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2} 
status=0 QTime=94734
2019-11-01 13:17:32.567 INFO  (qtp694316372-44688) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2} 
status=0 QTime=815338
2019-11-01 13:17:32.570 INFO  (qtp694316372-44688) [   ] 
o.a.s.s.HttpSolrCall Unable to write response, client closed connection 
or we are shutting down => org.eclipse.jetty.io.EofException: Closed

     at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)
org.eclipse.jetty.io.EofException: Closed


Jetty's EofException almost always means one specific thing.  The client 
closed the connection before Solr could respond, so when Solr finally 
finished processing and tried to have Jetty send the response, there was 
nowhere to send it -- the connection was gone.


The first two lines of the log snippet indicate that there was one 
COLSTATUS call that took nearly 95 seconds, and one that took 815 
seconds, which is close to 15 minutes.  Apparently those two calls 
completed at the same time, even though they did not start at the same 
time.  Which suggests that for many minutes, the Solr server has been 
under severe stress that prevented it from responding quickly to the 
COLSTATUS request.  The client gave up and closed its connection ... 
probably on the one that took almost 15 minutes.


I found that the COLSTATUS action is mentioned in the 8.1 reference 
guide on the "Collections API" page, but it is not there on the same 
page in the 8.2 guide.  That page in 8.2 appears to be significantly 
smaller and missing most of the documentation that 8.1 has.  So I think 
we had a problem with documentation generation on 8.2.


Based on what I can see in the response on the 8.1 guide, I'm betting 
that gathering COLSTATUS takes a fair amount of processing, and any 
performance issues will make it very slow.


Thanks,
Shawn

Re: Solr 8.2.0 - Unable to write response

[Joe Obernberger]

Thank you Shawn.

What I'm trying to get for my application is the commitTimeMSec. I use 
that value to build up an alias of solr collections.  Is there a better way?


-Joe

On 11/1/2019 10:17 AM, Shawn Heisey wrote:

On 11/1/2019 7:20 AM, Joe Obernberger wrote:
Hi All - getting this error from only one server in a 45 node cluster 
when calling COLSTATUS.  Any ideas?




2019-11-01 13:17:32.556 INFO (qtp694316372-44709) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=UNCLASS_2019_1_18_36&action=COLSTATUS&wt=javabin&version=2} 
status=0 QTime=94734
2019-11-01 13:17:32.567 INFO  (qtp694316372-44688) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={name=UNCLASS_2021_2_17_36&action=COLSTATUS&wt=javabin&version=2} 
status=0 QTime=815338
2019-11-01 13:17:32.570 INFO  (qtp694316372-44688) [   ] 
o.a.s.s.HttpSolrCall Unable to write response, client closed 
connection or we are shutting down => 
org.eclipse.jetty.io.EofException: Closed
 at 
org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:491)

org.eclipse.jetty.io.EofException: Closed


Jetty's EofException almost always means one specific thing.  The 
client closed the connection before Solr could respond, so when Solr 
finally finished processing and tried to have Jetty send the response, 
there was nowhere to send it -- the connection was gone.


The first two lines of the log snippet indicate that there was one 
COLSTATUS call that took nearly 95 seconds, and one that took 815 
seconds, which is close to 15 minutes.  Apparently those two calls 
completed at the same time, even though they did not start at the same 
time.  Which suggests that for many minutes, the Solr server has been 
under severe stress that prevented it from responding quickly to the 
COLSTATUS request.  The client gave up and closed its connection ... 
probably on the one that took almost 15 minutes.


I found that the COLSTATUS action is mentioned in the 8.1 reference 
guide on the "Collections API" page, but it is not there on the same 
page in the 8.2 guide.  That page in 8.2 appears to be significantly 
smaller and missing most of the documentation that 8.1 has.  So I 
think we had a problem with documentation generation on 8.2.


Based on what I can see in the response on the 8.1 guide, I'm betting 
that gathering COLSTATUS takes a fair amount of processing, and any 
performance issues will make it very slow.


Thanks,
Shawn