Re: How to get a solr core to persist

[Amanda Shuman]

Hi Shawn,

Thanks a million for your response! I really do appreciate it... this helps
clarify how things should be set up.

As for why things are set up the way they are and the webapps location...
honestly I think my predecessor did not really understand solr at all...
I'm trying to pick up the pieces now on the back-end. (On the bright side,
I did figure out how to modify the search relevance critieria in the config
files for our core, but I'm more of a front-end developer and that seemed a
lot more intuitive to me.)

It does seem that the solr home is currently in /var/solr/data (not
server/solr) because when I created a new core at command line, that's
where it went. We start/restart solr using /etc/init.d/ rather than
bin/solr.

If can ask a few very small follow-up questions to this:

"If you do a manual core creation, the core.properties file must NOT exist,
but the conf directory *must* exist with the proper contents."

1) so does this mean that on the back-end I should first create my new
core, e.g., core1 and then within that place a conf folder with all the
files? Same for the data folder? If so, is it fine to just use the existing
config files that I've previously worked on (i.e. the config for search
that I already modified)? I presume this won't be an issue.

2) does it matter if I create this core through the admin console or at
command line?

Thanks again!
Amanda

--
Dr. Amanda Shuman
Post-doc researcher, University of Freiburg, The Maoist Legacy Project

PhD, University of California, Santa Cruz
http://www.amandashuman.net/
http://www.prchistoryresources.org/
Office: +49 (0) 761 203 4925


On Tue, Nov 14, 2017 at 3:15 PM, Shawn Heisey  wrote:

> On 11/14/2017 2:14 AM, Amanda Shuman wrote:
>
>> We have just one solr core, which resides in the webapp foIder (in solr
>> 5.3.1 this is at /opt/solr-5.3.1/server/webapps/[corename]/  -- the data
>> folder is in the same place at /data).
>>
>
> Why is your core there?  That is not a typical location, and does explain
> the restart behavior you're seeing.
>
> Usually core directories go in the solr home.  If you start solr using
> "bin/solr start" directly without any options, the solr home will be in
> server/solr, not server/webapp.  How are you starting Solr?
>
> "Could not create a new core in
>> /opt/solr-5.3.1/server/webapps/[corename]/as another core is already
>> defined there"
>>
>
> When Solr starts, it begins searching the coreRootDirectory (which
> defaults to the solr home) for cores.  When it locates a core.properties
> file, that location becomes the instanceDir for a core.
>
> If you do a manual core creation, the core.properties file must NOT exist,
> but the conf directory *must* exist with the proper contents. The core
> creation will create that file.  If it already exists, then Solr will
> refuse to create the core, just as you have seen.
>
> The program directory location you have mentioned (/opt/solr-5.3.1) sounds
> like somebody did a service installation.  The default solr home when you
> install the service (and start Solr with /etc/init.d/ rather than
> bin/solr) is /var/solr/data.  This location can be overridden, but that's
> the default.
>
> Instead of having your core in webapp, move it to the solr home, wherever
> that is.  Then when you start Solr, it will find the core.
>
> If a service installation has been done, then you should not start Solr
> with "bin/solr" -- you should start the installed service.
>
> Thanks,
> Shawn
>

Re: How to get a solr core to persist

[Amanda Shuman]

Ah, also, this is what the admin console says for location of core docs
when I created the core at command line:

CWD:/opt/solr-5.3.1/serverInstance:/var/solr/data/[corename]Data:
/var/solr/data/[corename]/dataIndex:/var/solr/data/[corename]/data/index


--
Dr. Amanda Shuman
Post-doc researcher, University of Freiburg, The Maoist Legacy Project

PhD, University of California, Santa Cruz
http://www.amandashuman.net/
http://www.prchistoryresources.org/
Office: +49 (0) 761 203 4925


On Wed, Nov 15, 2017 at 10:28 AM, Amanda Shuman 
wrote:

> Hi Shawn,
>
> Thanks a million for your response! I really do appreciate it... this
> helps clarify how things should be set up.
>
> As for why things are set up the way they are and the webapps location...
> honestly I think my predecessor did not really understand solr at all...
> I'm trying to pick up the pieces now on the back-end. (On the bright side,
> I did figure out how to modify the search relevance critieria in the config
> files for our core, but I'm more of a front-end developer and that seemed a
> lot more intuitive to me.)
>
> It does seem that the solr home is currently in /var/solr/data (not
> server/solr) because when I created a new core at command line, that's
> where it went. We start/restart solr using /etc/init.d/ rather than
> bin/solr.
>
> If can ask a few very small follow-up questions to this:
>
> "If you do a manual core creation, the core.properties file must NOT
> exist, but the conf directory *must* exist with the proper contents."
>
> 1) so does this mean that on the back-end I should first create my new
> core, e.g., core1 and then within that place a conf folder with all the
> files? Same for the data folder? If so, is it fine to just use the existing
> config files that I've previously worked on (i.e. the config for search
> that I already modified)? I presume this won't be an issue.
>
> 2) does it matter if I create this core through the admin console or at
> command line?
>
> Thanks again!
> Amanda
>
> --
> Dr. Amanda Shuman
> Post-doc researcher, University of Freiburg, The Maoist Legacy Project
> 
> PhD, University of California, Santa Cruz
> http://www.amandashuman.net/
> http://www.prchistoryresources.org/
> Office: +49 (0) 761 203 4925 <+49%20761%202034925>
>
>
> On Tue, Nov 14, 2017 at 3:15 PM, Shawn Heisey  wrote:
>
>> On 11/14/2017 2:14 AM, Amanda Shuman wrote:
>>
>>> We have just one solr core, which resides in the webapp foIder (in solr
>>> 5.3.1 this is at /opt/solr-5.3.1/server/webapps/[corename]/  -- the data
>>> folder is in the same place at /data).
>>>
>>
>> Why is your core there?  That is not a typical location, and does explain
>> the restart behavior you're seeing.
>>
>> Usually core directories go in the solr home.  If you start solr using
>> "bin/solr start" directly without any options, the solr home will be in
>> server/solr, not server/webapp.  How are you starting Solr?
>>
>> "Could not create a new core in
>>> /opt/solr-5.3.1/server/webapps/[corename]/as another core is already
>>> defined there"
>>>
>>
>> When Solr starts, it begins searching the coreRootDirectory (which
>> defaults to the solr home) for cores.  When it locates a core.properties
>> file, that location becomes the instanceDir for a core.
>>
>> If you do a manual core creation, the core.properties file must NOT
>> exist, but the conf directory *must* exist with the proper contents. The
>> core creation will create that file.  If it already exists, then Solr will
>> refuse to create the core, just as you have seen.
>>
>> The program directory location you have mentioned (/opt/solr-5.3.1)
>> sounds like somebody did a service installation.  The default solr home
>> when you install the service (and start Solr with /etc/init.d/ rather
>> than bin/solr) is /var/solr/data.  This location can be overridden, but
>> that's the default.
>>
>> Instead of having your core in webapp, move it to the solr home, wherever
>> that is.  Then when you start Solr, it will find the core.
>>
>> If a service installation has been done, then you should not start Solr
>> with "bin/solr" -- you should start the installed service.
>>
>> Thanks,
>> Shawn
>>
>
>

Re: Solr 6.5.1 process crash after jshort_disjoint_arraycopy error

[tothis]

Hi Yago, 

we are facing the same problem. Could you solve it somehow? 

thx




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr 6.5.1 process crash after jshort_disjoint_arraycopy error

[Yago Riveiro]

Nop,

I never found a fix for this problem, sorry.

Regards.

--

Yago Riveiro

On 15 Nov 2017 09:44 +, tothis , wrote:
> Hi Yago,
>
> we are facing the same problem. Could you solve it somehow?
>
> thx
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Sol rCloud collection design considerations / best practice

[alessandro.benedetti]

"The main motivation is to support a geo-specific relevancy 
model which can easily be customized without stepping into each other"

Is your relevancy tuning massively index time based ?
i.e. will create massively different index content based on the geo location
?

If it is just query time based or lightly index based ( few fields of
difference across region), you don't need different collections at all to
have a customized relevancy model per use case.

In Solr you can define different request handlers with different query
parsers and search components specifications.
If you go in deep with relevancy tuning and for example you experiment
Learning To Rank, it supports passing the model name at query time, which
means you can use a different relevancy mode just passing it as a request
parameter.

Regards



-
---
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: How to routing document for send to particular shard range

[Ketan Thanki]

Thanks Erik,

I have re-mentions it as some required details are missing in my mail. Using 
CloudSolrClient my test case as below.
I have used routing key as projectId/2!documentId
 
1: Detail of Insert Document in SolrIndex
Document Size: 919551 
Document Batch Size for insert: 5000 documents in each thread
Total thread to Index: 184 (thread pool size 5)
With Routing: 251835 millisecond taken to index documents in index (Data index 
in 1 shard) 
Without Routing: 325108 millisecond taken to index documents in index (Data 
index in 4 shard)

2: Detail of search document/Retrieve from SolrIndex
Document Size: 919551 
Document Batch Size for retrieve: 1 (per query 1 document id used for 
search)
Total thread to Search: 93 (thread pool size 3)
Total Queries call: 93
With Routing: 94562 millisecond taken to search documents from index (Data in 1 
shard) 
Without Routing: 234242 millisecond taken to search documents From index (Data 
in 4 shard)

Retrieval query with parameter used fq & fl also.
 
So above shows my one model data which belongs to one project (also project is 
consisting of many models).
While refer above use-case my data in single shard which gives improvements 
with routing facility.
but when considering the production environment  If I have 800M documents (it 
may be more) in each of my 4 shard and considering above case so what would be 
better from below.
- to distribute project data in two shard
- to distribute project data in single shard

is there any documents limit for shard? And if my document  grow more than 
above mentions so is it causes any issue?

Please do needful.

Regards,
Ketan







 

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, November 14, 2017 9:33 PM
To: solr-user
Subject: Re: How to routing document for send to particular shard range

These numbers aren't very useful. inserting how much data? Querying how much 
data? What kinds of queries? Are you indexing in batches or one document at a 
time? Are you using SolrJ and CloudSolrClient?

94 seconds to do _what_? Execute 1,000 queries? Fetch all the documents from 
the shard? Execute one query?

What is "a huge amount of data"? I've seen 300M documents fit on one shard. 
I've seen people claim 1M documents is "huge".

Details matter. You might review:

https://wiki.apache.org/solr/UsingMailingLists

1:inserting data in 4 shard without document routing  time taken ( in
millisecond)  =325108
Inserting data in 1 shard with document routing time time taken ( in
millisecond)  =251835

2: retrieving data from 4 shard without document routing time taken( in 
millisecond)  = 234242 And retrieving data from 1 shard with document routing 
time taken ( in millisecond)= 94562

Best,
Erick

On Tue, Nov 14, 2017 at 6:50 AM, Ketan Thanki  wrote:
> Thanks Amrit ,
>
> Actually we have huge amount of data so that's why thinking to index data 
> into particular shard accept it's looks difficult but need to achieve the 
> performance using document routing for huge data.
>
> With configuration of  4 shard and 4 replica  is it better to distribute the 
> one project data in multiple shard or in one shard which one is  feasible 
> using document routing because needs the best performance while insertion & 
> retrieval of document. And there would be the multiple projects of client 
> which has huge amount of data.
>
> I also taken the reading with 4 shard and 4 replica where without routing 
> data are distribute among all 4 shard  and with routing its distributes in 1 
> shard because of used 1 bit of shard key like projectId/1!DocumentId.my 
> reading looks as below.
> 1:inserting data in 4 shard without document routing  time taken ( in 
> millisecond)  =325108 Inserting data in 1 shard with document routing 
> time time taken ( in millisecond)  =251835
>
> 2: retrieving data from 4 shard without document routing time taken( 
> in millisecond)  = 234242 And retrieving data from 1 shard with 
> document routing time taken ( in millisecond)= 94562
>
> As per above reading getting  performance in local  while data in 1 shard but 
> in production there will be huge data so is it need to distribute in 2 shard 
> or in 1 shard which one is feasible for achieve better performance.
>
>
> Regards,
> Ketan
>
> -Original Message-
> From: Amrit Sarkar [mailto:sarkaramr...@gmail.com]
> Sent: Monday, November 13, 2017 8:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: How to routing document for send to particular shard 
> range
>
> Surely someone else can chim in;
>
> but when you say: "so regarding to it we need to index the particular
>> client data into particular shard so if its  manageable than we will 
>> improve the performance as we need"
>
>
> You can / should create different collections for different client data, so 
> that you can for surely improve performance as per need. There are multiple 
> configurations which drives indexing and querying capabilities and 
> inco

Analyse Fieldname API

[kumar gaurav]

Hi

Solr has panel to Analyse Fieldname i.e.

http://localhost:8983/solr/#/corename/analysis

I need an API which will return analysis information in JSON format like
search handler .

 Someone ! Is there any API regarding the same ?

Thanks in advance :)

Re: How to get a solr core to persist

[Shawn Heisey]

On 11/15/2017 2:28 AM, Amanda Shuman wrote:

1) so does this mean that on the back-end I should first create my new
core, e.g., core1 and then within that place a conf folder with all the
files? Same for the data folder? If so, is it fine to just use the existing
config files that I've previously worked on (i.e. the config for search
that I already modified)? I presume this won't be an issue.

2) does it matter if I create this core through the admin console or at
command line?


You can create your cores however you like.  I actually create all my 
cores completely by hand, including the core.properties file, and let 
Solr discover them on startup.  Mostly I just copy an existing core, 
change core.properties to correct values, make any config changes I 
need, and restart Solr.


If you want to use the admin UI (or the CoreAdmin API directly, which is 
what the admin UI calls), then the instanceDir must have a conf 
directory with all the config files you require for the core, and NOT 
have a core.properties file.  If you're adding a core that already has a 
an index, then you would also include the data directory in the core's 
instanceDir.  If you want to relocate the data, you can add a dataDir 
property to core.properties.  If it has a relative path, it is relative 
to the core.properties location.


The commandline creation works pretty well.  The way it works is by 
copying a configset (which may be in server/solr/configsets or in a 
custom location) to the "conf" directory in the core, then calling the 
CoreAdmin API to actually add the core to Solr (and create 
core.properties so it'll get picked up on restart).


Thanks,
Shawn

TimeZone issue

[Renuka Srishti]

Hello All,

I am working on CSV export using Apache Solr. I have written all the
required query and set wt as CSV. I am getting my results as I want,but the
problem is TimeZone.

Solr stores date value in UTC, but my client timeZone is different. Is
there any way to convert date timeZone from UTC to clientTimeZone direclty
in the Solr response?

When I extract the date from the Solr response, I can convert it into
clientTimeZone by implementing a custom method. But I want the date in
client TimeZone direclty in the solr response.

Thanks
Renuka Srishti

Re: TimeZone issue

[Shawn Heisey]

On 11/15/2017 5:34 AM, Renuka Srishti wrote:

I am working on CSV export using Apache Solr. I have written all the
required query and set wt as CSV. I am getting my results as I want,but the
problem is TimeZone.

Solr stores date value in UTC, but my client timeZone is different. Is
there any way to convert date timeZone from UTC to clientTimeZone direclty
in the Solr response?


Not that I know of.  UTC is the only storage/transfer method that works 
in all situations.  Converting dates to the local timezone is a task for 
the client, when it displays the date to a user.


Typically, you would consume the response from Solr into object types 
for the language your application is written in.  A date value in the 
response should end up in a date object.  Date objects in most 
programming languages have the ability to display in specific timezones.


Thanks,
Shawn

Re: Analyse Fieldname API

[Erik Hatcher]

Turn on your browsers developer mode and check out the HTTP requests behind the 
scenes of that page.   

Yes!   ;)

> On Nov 15, 2017, at 07:19, kumar gaurav  wrote:
> 
> Hi
> 
> Solr has panel to Analyse Fieldname i.e.
> 
> http://localhost:8983/solr/#/corename/analysis
> 
> I need an API which will return analysis information in JSON format like
> search handler .
> 
> Someone ! Is there any API regarding the same ?
> 
> Thanks in advance :)

Applying boosts inside a SpanQuery

[Antoine ORY-LAMBALLE]

Hello everybody,

First of all, thanks for this great tool!

Do you know if there is a way to apply a *boost inside a SpanOrQuery*
please? (Solr 7.1, lucene of same version I think.)

Example of query structure generated by edismax query parser (inside the
usual BooleanQuery / DisjunctionMaxQuery for multiple fields):

   - SpanNearQuery
  - List of SpanOrQuery
 - List of SpanTermQuery or SpanNearQuery* <-- I want to boost
 those terms*

I want to apply a boost to each clause inside the SpanOrQuery. So I tried
(via custom query parser extending edismax):

   - SpanNearQuery
  - List of SpanOrQuery
 - List of *SpanBoostQuery*, each wrapping:
- a SpanTermQuery or a SpanNearQuery.

The boost seems to be ignored, although the query is executed successfully
(except boost not applied).

Here is the use case:

The input is a sentence, a PhraseQuery to valuate proximity. I am using
edismax query parser and a SynonymGraphFilter and I want to apply a
different boost for each synonym, so the boost information is attached to
each term in the synonym file (e.g. 0.7_foo). In case of multi-term
synonyms (expanded version), edismax generates a SpanQuery, which sounds to
me like a graph search, so far so good.

Where it fails is when I insert a SpanBoostQuery to wrap each clause inside
the SpanOrQuery (via a custom query parser extending edismax, extracting
boosts from term text). While the query is still returning results, the
boost seems just ignored.



Is that a misusage? A bug? Any advice about how I can fix it or work around
it please?

To have a similar behavior, I replaced the SpanQuery by a list of
PhraseQueries, one for each possible combination of terms in the SpanQuery,
which can result in lots of phrases, and the performances seem highly
affected.

Thanks a lot!
Best Regards,
Antoine

Re: How to get a solr core to persist

[Rick Leir]

Hi Shawn, Amanda
When we put the data under /var/lib, I feel a need to put the config under 
/etc. Is this recommended, and would you use a symbolic link for the conf dir?
Cheers--Rick
-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: Limiting by range of sum across documents

[chris]

Emir,
It certainly seems like I'll need to use streaming expressions.
Thanks for your help!
Chris


> Hi Chris,

> I misunderstood your requirement. I am not aware of some facet result 
> filtering feature. What you could do is sort facet results by sum and load 
> page by page but that does not sound like a good solution. Did you try using 
> streaming expressions - I don’t have much experience with this
feature so would have to play a bit before giving answer if possible and how to 
do it, but I guess someone will be able to give some pointers.
>

> Thanks,

> Emir

> --

> Monitoring - Log Management - Alerting - Anomaly Detection

> Solr & Elasticsearch Consulting Support Training - http://sematext.com/

>

>

>

>> On 14 Nov 2017, at 16:51, ch...@yeeplusplus.com wrote:

>>

>>

>>

>>

>> I'm not looking for products where the price is in the range [100, 200].

>> I'm looking for manufacturers for which the sum of the prices of all of 
>> their products is in the range [100, 200].

>> �

>>

>>

>>> Hi Chris,

>>

>>>

>>

>>> I assumed that you apply some sort of fq=price:[100 TO 200] to focus on 
>>> wanted products.

>>

>>>

>>

>>> Can you share full json faceting request - numFound:0 suggest that 
>>> something is completely wrong.

>>

>>>

>>

>>> Thanks,

>>

>>> Emir

>>

>>> --

>>

>>> Monitoring - Log Management - Alerting - Anomaly Detection

>>

>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/

>>

>>>

>>

>>>

>>

>>>

>>

 On 13 Nov 2017, at 21:56, ch...@yeeplusplus.com wrote:

>>



>>



>>



>>



>>

 �

>>

 Hi Emir,

>>

 I can't apply filters to the original query because I don't know in 
 advance which filters will meet the criterion I'm looking for.� Unless I'm 
 missing something obvious.�

>>

 �

>>

 I tried the JSON facet you suggested but received

>>



>>

 "response":{"numFound":0,"start":0,"maxScore":0.0,"docs":[]

>>



>>

 � },

>>



>>

 � "facet_counts":{

>>



>>

 � � "facet_queries":{},

>>



>>

 � � "facet_fields":{},

>>



>>

 � � "facet_dates":{},

>>



>>

 � � "facet_ranges":{},

>>



>>

 � � "facet_intervals":{},

>>



>>

 � � "facet_heatmaps":{}},

>>



>>

 � "facets":{

>>



>>

 � � "count":0}}

>>



>>

 �

>>



>>

 �

>>



>>



>>

> Hi Chris,

>>



>>

> You mention it returns all manufacturers? Even after you apply filters 
> (don’t see filter in your example)? You can control how many facets 
> are returned with facet.limit and you can use face.pivot.mincount to 
> determine how many facets are returned. If you calculate sum
on
>> all

 manufacturers, it can last.

>>

>

>>



>>

> Maybe you can try json faceting. Something like (url style):

>>



>>

>

>>



>>

> …&json.facet={sumByManu:{terms:{field:manufacturer,facet:{sum:”sum(price)”

>>



>>

>

>>



>>

> HTH,

>>



>>

> Emir

>>



>>

> --

>>



>>

> Monitoring - Log Management - Alerting - Anomaly Detection

>>



>>

> Solr & Elasticsearch Consulting Support Training - http://sematext.com/

>>



>>

>

>>



>>

>

>>



>>

>

>>



>>

>> On 12 Nov 2017, at 19:09, ch...@yeeplusplus.com wrote:

>>



>>

>>

>>



>>

>>

>>



>>

>>

>>



>>

>>

>>



>>

>> I have documents in solr that look like this:

>>



>>

>> {

>>



>>

>> "id": "acme-1",

>>



>>

>> "manufacturer": "acme",

>>



>>

>> "product_name": "Foo",

>>



>>

>> "price": 3.4

>>



>>

>> }

>>



>>

>>

>>



>>

>> There are about

>>



>>

>> 150,000 manufacturers, each of which have between 20,000 and 1,000,000 
>> products.

>>



>>

>> I'd like to return the sum of all prices that are in the range [100, 
>> 200], faceted by manufacturer. In other words, for each manufacturer, 
>> sum the prices of all products for that manufacturer,

>>



>>

>> and return the sum and the manufacturer name. For example:

>>



>>

>> [

>>



>>

>> {

>>



>>

>> "manufacturer": "acme",

>>



>>

>> "sum": 150.5

>>



>>

>> },

>>



>>

>> {

>>



>>

>> "manufacturer": "Johnson,

>>



>>

>> Inc.",

>>



>>

>> "sum": 167.0

>>



>>

>> },

>>



>>

>> ...

>>



>>

>> ]

>>



>>

>>

>>



>>

>> I tried this:

>>



>>

>> q=*:*&rows=0&stats=true&stats.field={!tag=piv1 
>> sum=true}price&facet=true&facet.pivot={!stats=piv1}manufacturer

>>



>>

>> which "works" on a test

>>



>>

>> subset of 1,000 manufacturers. However, there are two problems:

>>



>>

>

Re: How to get a solr core to persist

[Erick Erickson]

Well, if you must ;)

Basically whatever is easier. "Core discovery" starts at SOLR_HOME,
wherever you've defined that. Which can be done with a sysvar
"-Dwhatrever" at start time or the "-s" param if using the bin/solr
start scripts. That's the root for finding all "core.properties"
files. Each core.properties file defines a core.

_within_ a core.properties file you can specify a dataDir pointing
wherever you want which is where the index goes.

If using the core admin API to create a core, you can specify dataDir
as one of the params.

Really, about all the core admin CREATE API does is create the
core.properties file, so Shawn's process of setting it all up
_including_ a core.properties file and starting Solr works perfectly.

The only caution when specifying dataDir for a particular core is it
as A Very Bad Thing if two cores point to the _same_ dataDir. One of
them will refuse to load due to the index being locked.

A lot of this is at:
https://lucene.apache.org/solr/guide/6_6/defining-core-properties.html

Best,
Erick

On Wed, Nov 15, 2017 at 5:05 AM, Rick Leir  wrote:
> Hi Shawn, Amanda
> When we put the data under /var/lib, I feel a need to put the config under 
> /etc. Is this recommended, and would you use a symbolic link for the conf dir?
> Cheers--Rick
> --
> Sorry for being brief. Alternate email is rickleir at yahoo dot com

Re: Error when indexing EML files in Solr 7.1.0

[Karthik Ramachandran]

JIRA already exists, https://issues.apache.org/jira/browse/SOLR-11622.


On Mon, Nov 13, 2017 at 5:55 PM, Zheng Lin Edwin Yeo 
wrote:

> Hi Erick,
>
> I have added the apache-mime4j-core-0.7.2.jar in the Java Build Path of the
> Eclipse, but it is also not working.
>
> Regards,
> Edwin
>
> On 13 November 2017 at 23:33, Erick Erickson 
> wrote:
>
> > Where are you getting your mime4j file? MimeConfig is in
> > /extraction/lib/apache-mime4j-core-0.7.2.jar and you need to make sure
> > you're including that at a guess.
> >
> > Best,
> > Erick
> >
> > On Mon, Nov 13, 2017 at 6:15 AM, Zheng Lin Edwin Yeo
> >  wrote:
> > > Hi,
> > >
> > > I am using Solr 7.1.0, and I am trying to index EML files using the
> > > SimplePostTools.
> > >
> > > However, I get the following error
> > >
> > > java.lang.NoClassDefFoundError:
> > > org/apache/james/mime4j/stream/MimeConfig$Builder
> > >
> > >
> > > Is there any we class or dependencies which I need to add as compared
> to
> > > Solr 6?
> > >
> > > The indexing is ok for other files type like .doc, .ppt. I only face
> the
> > > error when indexing .eml files.
> > >
> > > Regards,
> > > Edwin
> >
>

Re: [WARNING: DropBox links may be malicious.] Re: Admin Console Question

[Webster Homer]

I found that my boss's solr admin console did display the Args the only
install I have that does...
I do see errors in both Consoles. I see more errors on the ones that don't
display Args
Here are the errors that only show up when Args doesn't:
Error: [ngRepeat:dupes] Duplicates in a repeater are not allowed. Use
'track by' expression to specify unique keys. Repeater: arg in
commandLineArgs, Duplicate key: string:-XX:+UseGCLogFileRotation, Duplicate
value: -XX:+UseGCLogFileRotation
http://errors.angularjs.org/1.3.8/ngRepeat/dupes?p0=arg%20in%20commandLineArgs&p1=string%3A-XX%3A%2BUseGCLogFileRotation&p2=-XX%3A%2BUseGCLogFileRotation
at angular.js:86
at ngRepeatAction (angular.js:24506)
at Object.$watchCollectionAction [as fn] (angular.js:14115)
at Scope.$digest (angular.js:14248)
at Scope.$apply (angular.js:14511)
at done (angular.js:9669)
at completeRequest (angular.js:9859)
at XMLHttpRequest.requestLoaded (angular.js:9800)
(anonymous) @ angular.js:11617


This is all of the errors I see when loading the page. Most of these show
up when loading the page

angular.js:11617 TypeError: Cannot read property 'default_text' of undefined
at initOrUpdate (angular-chosen.js:80)
at NgModelController.ngModel.$render (angular-chosen.js:95)
at Object.ngModelWatch (angular.js:20998)
at Scope.$digest (angular.js:14240)
at Scope.$apply (angular.js:14511)
at bootstrapApply (angular.js:1472)
at Object.invoke (angular.js:4205)
at doBootstrap (angular.js:1470)
at bootstrap (angular.js:1490)
at angularInit (angular.js:1384)
(anonymous) @ angular.js:11617
(anonymous) @ angular.js:8567
$digest @ angular.js:14266
$apply @ angular.js:14511
bootstrapApply @ angular.js:1472
invoke @ angular.js:4205
doBootstrap @ angular.js:1470
bootstrap @ angular.js:1490
angularInit @ angular.js:1384
(anonymous) @ angular.js:26088
j @ jquery-2.1.3.min.js:27
fireWith @ jquery-2.1.3.min.js:27
ready @ jquery-2.1.3.min.js:27
I @ jquery-2.1.3.min.js:27
2angular.js:11617 TypeError: Cannot read property 'results_none_found' of
undefined
at disableWithMessage (angular-chosen.js:89)
at angular-chosen.js:123
at angular.js:16228
at completeOutstandingRequest (angular.js:4925)
at angular.js:5305
(anonymous) @ angular.js:11617
(anonymous) @ angular.js:8567
(anonymous) @ angular.js:16231
completeOutstandingRequest @ angular.js:4925
(anonymous) @ angular.js:5305
setTimeout (async)
Browser.self.defer @ angular.js:5303
timeout @ angular.js:16226
(anonymous) @ angular-chosen.js:114
$watchCollectionAction @ angular.js:14113
$digest @ angular.js:14248
$apply @ angular.js:14511
bootstrapApply @ angular.js:1472
invoke @ angular.js:4205
doBootstrap @ angular.js:1470
bootstrap @ angular.js:1490
angularInit @ angular.js:1384
(anonymous) @ angular.js:26088
j @ jquery-2.1.3.min.js:27
fireWith @ jquery-2.1.3.min.js:27
ready @ jquery-2.1.3.min.js:27
I @ jquery-2.1.3.min.js:27
ngtimeago.js:92 about 17 hoursago
angular.js:11617 Error: [ngRepeat:dupes] Duplicates in a repeater are not
allowed. Use 'track by' expression to specify unique keys. Repeater: arg in
commandLineArgs, Duplicate key: string:-Xss256k, Duplicate value: -Xss256k
http://errors.angularjs.org/1.3.8/ngRepeat/dupes?p0=arg%20in%20commandLineArgs&p1=string%3A-Xss256k&p2=-Xss256k
at angular.js:86
at ngRepeatAction (angular.js:24506)
at Object.$watchCollectionAction [as fn] (angular.js:14115)
at Scope.$digest (angular.js:14248)
at Scope.$apply (angular.js:14511)
at done (angular.js:9669)
at completeRequest (angular.js:9859)
at XMLHttpRequest.requestLoaded (angular.js:9800)
(anonymous) @ angular.js:11617
(anonymous) @ angular.js:8567
$digest @ angular.js:14266
$apply @ angular.js:14511
done @ angular.js:9669
completeRequest @ angular.js:9859
requestLoaded @ angular.js:9800
XMLHttpRequest.send (async)
(anonymous) @ angular.js:9837
sendReq @ angular.js:9638
serverRequest @ angular.js:9354
processQueue @ angular.js:13193
(anonymous) @ angular.js:13209
$eval @ angular.js:14406
$digest @ angular.js:14222
$apply @ angular.js:14511
done @ angular.js:9669
completeRequest @ angular.js:9859
requestLoaded @ angular.js:9800
XMLHttpRequest.send (async)
(anonymous) @ angular.js:9837
sendReq @ angular.js:9638
serverRequest @ angular.js:9354
processQueue @ angular.js:13193
(anonymous) @ angular.js:13209
$eval @ angular.js:14406
$digest @ angular.js:14222
$apply @ angular.js:14511
bootstrapApply @ angular.js:1472
invoke @ angular.js:4205
doBootstrap @ angular.js:1470
bootstrap @ angular.js:1490
angularInit @ angular.js:1384
(anonymous) @ angular.js:26088
j @ jquery-2.1.3.min.js:27
fireWith @ jquery-2.1.3.min.js:27
ready @ jquery-2.1.3.min.js:27
I @ jquery-2.1.3.min.js:27

On Tue, Nov 14, 2017 at 7:12 PM, Rick Leir  wrote:

> Homer
> In chrome, right-click and choose 'inspect' at the bottom. Now go to the
> network tab then reload the page. Are you seeing errors? Tell!
> Thanks
> Rick
>
> On November 14, 2017 3:14:

Re: [WARNING: DropBox links may be malicious.] Re: Admin Console Question

[Cassandra Targett]

So, from looking at those errors + a bit of Googling, it's complaining that
there are duplicate values in the Args list:

- Repeater: arg in commandLineArgs, Duplicate key:
string:-XX:+UseGCLogFileRotation,
Duplicate
value: -XX:+UseGCLogFileRotation
- Repeater: arg in commandLineArgs, Duplicate key: string:-Xss256k,
Duplicate value: -Xss256k

This tells us a bit about what is happening (the UI finds duplicates in the
arguments), but not why you are the only one who sees this.

>From what I understand, all the UI does is make a call to
http://localhost:8983/solr/admin/info/system and parse the JSON response in
various ways. The Args section comes from the "jvm.jmx.commandLineArgs"
section of that. Somewhere maybe that data is being requested twice and
making a duplicate set of data for the UI to parse?

What do you see when you make a direct call to those stats (
http://localhost:8983/solr/admin/info/system) in your browser? Are they
duplicated? Any errors in the logs?

Unfortunately, these are only clues - maybe they will help someone take
this a step further. If you can, you may also try another browser to see if
it occurs there also.

On Wed, Nov 15, 2017 at 9:40 AM, Webster Homer 
wrote:

> I found that my boss's solr admin console did display the Args the only
> install I have that does...
> I do see errors in both Consoles. I see more errors on the ones that don't
> display Args
> Here are the errors that only show up when Args doesn't:
> Error: [ngRepeat:dupes] Duplicates in a repeater are not allowed. Use
> 'track by' expression to specify unique keys. Repeater: arg in
> commandLineArgs, Duplicate key: string:-XX:+UseGCLogFileRotation,
> Duplicate
> value: -XX:+UseGCLogFileRotation
> http://errors.angularjs.org/1.3.8/ngRepeat/dupes?p0=arg%
> 20in%20commandLineArgs&p1=string%3A-XX%3A%2BUseGCLogFileRotation&p2=-XX%
> 3A%2BUseGCLogFileRotation
> at angular.js:86
> at ngRepeatAction (angular.js:24506)
> at Object.$watchCollectionAction [as fn] (angular.js:14115)
> at Scope.$digest (angular.js:14248)
> at Scope.$apply (angular.js:14511)
> at done (angular.js:9669)
> at completeRequest (angular.js:9859)
> at XMLHttpRequest.requestLoaded (angular.js:9800)
> (anonymous) @ angular.js:11617
>
>
> This is all of the errors I see when loading the page. Most of these show
> up when loading the page
>
> angular.js:11617 TypeError: Cannot read property 'default_text' of
> undefined
> at initOrUpdate (angular-chosen.js:80)
> at NgModelController.ngModel.$render (angular-chosen.js:95)
> at Object.ngModelWatch (angular.js:20998)
> at Scope.$digest (angular.js:14240)
> at Scope.$apply (angular.js:14511)
> at bootstrapApply (angular.js:1472)
> at Object.invoke (angular.js:4205)
> at doBootstrap (angular.js:1470)
> at bootstrap (angular.js:1490)
> at angularInit (angular.js:1384)
> (anonymous) @ angular.js:11617
> (anonymous) @ angular.js:8567
> $digest @ angular.js:14266
> $apply @ angular.js:14511
> bootstrapApply @ angular.js:1472
> invoke @ angular.js:4205
> doBootstrap @ angular.js:1470
> bootstrap @ angular.js:1490
> angularInit @ angular.js:1384
> (anonymous) @ angular.js:26088
> j @ jquery-2.1.3.min.js:27
> fireWith @ jquery-2.1.3.min.js:27
> ready @ jquery-2.1.3.min.js:27
> I @ jquery-2.1.3.min.js:27
> 2angular.js:11617 TypeError: Cannot read property 'results_none_found' of
> undefined
> at disableWithMessage (angular-chosen.js:89)
> at angular-chosen.js:123
> at angular.js:16228
> at completeOutstandingRequest (angular.js:4925)
> at angular.js:5305
> (anonymous) @ angular.js:11617
> (anonymous) @ angular.js:8567
> (anonymous) @ angular.js:16231
> completeOutstandingRequest @ angular.js:4925
> (anonymous) @ angular.js:5305
> setTimeout (async)
> Browser.self.defer @ angular.js:5303
> timeout @ angular.js:16226
> (anonymous) @ angular-chosen.js:114
> $watchCollectionAction @ angular.js:14113
> $digest @ angular.js:14248
> $apply @ angular.js:14511
> bootstrapApply @ angular.js:1472
> invoke @ angular.js:4205
> doBootstrap @ angular.js:1470
> bootstrap @ angular.js:1490
> angularInit @ angular.js:1384
> (anonymous) @ angular.js:26088
> j @ jquery-2.1.3.min.js:27
> fireWith @ jquery-2.1.3.min.js:27
> ready @ jquery-2.1.3.min.js:27
> I @ jquery-2.1.3.min.js:27
> ngtimeago.js:92 about 17 hoursago
> angular.js:11617 Error: [ngRepeat:dupes] Duplicates in a repeater are not
> allowed. Use 'track by' expression to specify unique keys. Repeater: arg in
> commandLineArgs, Duplicate key: string:-Xss256k, Duplicate value: -Xss256k
> http://errors.angularjs.org/1.3.8/ngRepeat/dupes?p0=arg%
> 20in%20commandLineArgs&p1=string%3A-Xss256k&p2=-Xss256k
> at angular.js:86
> at ngRepeatAction (angular.js:24506)
> at Object.$watchCollectionAction [as fn] (angular.js:14115)
> at Scope.$digest (angular.js:14248)
> at Scope.$apply (angular.js:14511)
> at done (angular.js:9669)
> at completeRequest (angular.js:9859)
> 

Re: Analyse Fieldname API

[kumar gaurav]

I got it Erik

here is API

http://localhost:8983/solr/core/analysis/field?wt=json&analysis.showmatch=false&analysis.fieldvalue=nokia%20mobile&analysis.fieldname=productfield

thank you so much :)



On Wed, Nov 15, 2017 at 6:18 PM, Erik Hatcher 
wrote:

> Turn on your browsers developer mode and check out the HTTP requests
> behind the scenes of that page.
>
> Yes!   ;)
>
> > On Nov 15, 2017, at 07:19, kumar gaurav  wrote:
> >
> > Hi
> >
> > Solr has panel to Analyse Fieldname i.e.
> >
> > http://localhost:8983/solr/#/corename/analysis
> >
> > I need an API which will return analysis information in JSON format like
> > search handler .
> >
> > Someone ! Is there any API regarding the same ?
> >
> > Thanks in advance :)
>

Re: [WARNING: DropBox links may be malicious.] Re: [WARNING: DropBox links may be malicious.] Re: Admin Console Question

[Webster Homer]

I see the same issue with Firefox so it's not strictly browser dependent. I
also have one installation that doesn't have the problem.

The JSON in the endpoint clearly has some duplicates
"jmx":{

"bootclasspath":"/usr/java/jdk1.8.0_51/jre/lib/resources.jar:/usr/java/jdk1.8.0_51/jre/lib/rt.jar:/usr/java/jdk1.8.0_51/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_51/jre/lib/jsse.jar:/usr/java/jdk1.8.0_51/jre/lib/jce.jar:/usr/java/jdk1.8.0_51/jre/lib/charsets.jar:/usr/java/jdk1.8.0_51/jre/lib/jfr.jar:/usr/java/jdk1.8.0_51/jre/classes",

"classpath":"/usr/local/solr/solr-7.1.0/server/lib/gmetric4j-1.0.7.jar:/usr/local/solr/solr-7.1.0/server/lib/javax.servlet-api-3.1.0.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-continuation-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-deploy-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-http-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-io-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-jmx-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-rewrite-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-security-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-server-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-servlet-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-servlets-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-util-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-webapp-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/jetty-xml-9.3.20.v20170531.jar:/usr/local/solr/solr-7.1.0/server/lib/metrics-core-3.2.2.jar:/usr/local/solr/solr-7.1.0/server/lib/metrics-ganglia-3.2.2.jar:/usr/local/solr/solr-7.1.0/server/lib/metrics-graphite-3.2.2.jar:/usr/local/solr/solr-7.1.0/server/lib/metrics-jetty9-3.2.2.jar:/usr/local/solr/solr-7.1.0/server/lib/metrics-jvm-3.2.2.jar:/usr/local/solr/solr-7.1.0/server/lib/ext/jcl-over-slf4j-1.7.7.jar:/usr/local/solr/solr-7.1.0/server/lib/ext/jul-to-slf4j-1.7.7.jar:/usr/local/solr/solr-7.1.0/server/lib/ext/log4j-1.2.17.jar:/usr/local/solr/solr-7.1.0/server/lib/ext/slf4j-api-1.7.7.jar:/usr/local/solr/solr-7.1.0/server/lib/ext/slf4j-log4j12-1.7.7.jar:/usr/local/solr/solr-7.1.0/server/resources",
  "commandLineArgs":["-Xms512m",
"-Xmx512m",
"-XX:NewRatio=3",
"-XX:SurvivorRatio=4",
"-XX:TargetSurvivorRatio=90",
"-XX:MaxTenuringThreshold=8",
"-XX:+UseConcMarkSweepGC",
"-XX:+UseParNewGC",
"-XX:ConcGCThreads=4",
"-XX:ParallelGCThreads=4",
"-XX:+CMSScavengeBeforeRemark",
"-XX:PretenureSizeThreshold=64m",
"-XX:+UseCMSInitiatingOccupancyOnly",
"-XX:CMSInitiatingOccupancyFraction=50",
"-XX:CMSMaxAbortablePrecleanTime=6000",
"-XX:+CMSParallelRemarkEnabled",
"-XX:+ParallelRefProcEnabled",
"-verbose:gc",
"-XX:+PrintHeapAtGC",
"-XX:+PrintGCDetails",
"-XX:+PrintGCDateStamps",
"-XX:+PrintGCTimeStamps",
"-XX:+PrintTenuringDistribution",
"-XX:+PrintGCApplicationStoppedTime",
"-Xloggc:/var/solr1/logs/solr_gc.log",
"-XX:+UseGCLogFileRotation",
"-XX:NumberOfGCLogFiles=9",
"-XX:GCLogFileSize=20M",
"-DzkClientTimeout=15000",
"-DzkHost=localhost:2181/solr",
"-Dsolr.log.dir=/var/solr1/logs",
"-Djetty.port=8983",
"-DSTOP.PORT=7983",
"-DSTOP.KEY=solrrocks",
"-Duser.timezone=UTC",
"-Djetty.home=/usr/local/solr/solr1/server",
"-Dsolr.solr.home=/var/solr1/data",
"-Dsolr.data.home=",
"-Dsolr.install.dir=/usr/local/solr/solr1",

"-Dsolr.default.confdir=/usr/local/solr/solr1/server/solr/configsets/_default/conf",
"-Dlog4j.configuration=file:/var/solr1/log4j.properties",
"-Xss256k",
"-Xss256k",
"-Dsolr.jetty.https.port=8983",
"-Dsolr.log.muteconsole",
"-XX:OnOutOfMemoryError=/usr/local/solr/solr1/bin/oom_solr.sh 8983
/var/solr1/logs"],
  "startTime":"2017-11-14T22:27:58.591Z",
  "upTimeMS":69369776}},

On Wed, Nov 15, 2017 at 10:33 AM, Cassandra Targett 
wrote:

> So, from looking at those errors + a bit of Googling, it's complaining that
> there are duplicate values in the Args list:
>
> - Repeater: arg in commandLineArgs, Duplicate key:
> string:-XX:+UseGCLogFileRotation,
> Duplicate
> value: -XX:+UseGCLogFileRotation
> - Repeater: arg in commandLineArgs, Duplicate key: string:-Xss256k,
> Duplicate value: -Xss256k
>
> This tells us a bit about what is happening (the UI finds duplicates in the
> arguments), but not why you are the only one who sees this.
>
> From what I understand, all the UI does is make a call to
> http://localhost:8983/solr/admin/info/system and parse the JSON response
> in
> various ways. The Args section comes from the "jvm.jmx.commandLineArgs"
> section of that. Somewhere maybe that data is being requested

Re: Admin Console Question

[Shawn Heisey]

On 11/15/2017 8:40 AM, Webster Homer wrote:
> I do see errors in both Consoles. I see more errors on the ones that don't
> display Args
> Here are the errors that only show up when Args doesn't:
> Error: [ngRepeat:dupes] Duplicates in a repeater are not allowed. Use
> 'track by' expression to specify unique keys. Repeater: arg in
> commandLineArgs, Duplicate key: string:-XX:+UseGCLogFileRotation, Duplicate
> value: -XX:+UseGCLogFileRotation

> angular.js:11617 Error: [ngRepeat:dupes] Duplicates in a repeater are not
> allowed. Use 'track by' expression to specify unique keys. Repeater: arg in
> commandLineArgs, Duplicate key: string:-Xss256k, Duplicate value: -Xss256k
>

This was the clue I needed.

I added this line to the end of solr.in.cmd (I'm doing this testing on
Windows):

set SOLR_OPTS=%SOLR_OPTS% -Xss256k

With that change and a Solr restart, the Args information disappeared
from the admin UI.

Somewhere, likely in your include script, you have defined custom
arguments that have duplicated the -Xss256k and
-XX:+UseGCLogFileRotation arguments that are included by default.  There
may be other duplicates, but those are the ones that were included in
the error information you shared.  If you adjust the startup
configuration so that there are no duplicate commandline arguments, then
restart Solr, it should display.

This does mean that Solr has a bug in the admin UI, but it's one that
you can work around by removing duplicate arguments.  The angular code
used for the argument display cannot handle duplicate entries.  Here's
the issue I created for the problem:

https://issues.apache.org/jira/browse/SOLR-11645

There's a patch attached to the issue that fixes the problem for me, and
some instructions for fixing up a binary download with that change
rather than a source checkout.

Thanks,
Shawn

Re: Admin Console Question

[Webster Homer]

In the solr.in.sh script I do see this:
# Set the thread stack size
SOLR_OPTS="$SOLR_OPTS -Xss256k"

I don't remember ever changing this, but it's only there once

I can't find a reference to +UseGCLogFileRotation at all.

I don't see anyplace where we set either of these twice.

We were migrating from solr 6.2.0 if that makes any difference

On Wed, Nov 15, 2017 at 12:55 PM, Shawn Heisey  wrote:

> On 11/15/2017 8:40 AM, Webster Homer wrote:
> > I do see errors in both Consoles. I see more errors on the ones that
> don't
> > display Args
> > Here are the errors that only show up when Args doesn't:
> > Error: [ngRepeat:dupes] Duplicates in a repeater are not allowed. Use
> > 'track by' expression to specify unique keys. Repeater: arg in
> > commandLineArgs, Duplicate key: string:-XX:+UseGCLogFileRotation,
> Duplicate
> > value: -XX:+UseGCLogFileRotation
> 
> > angular.js:11617 Error: [ngRepeat:dupes] Duplicates in a repeater are not
> > allowed. Use 'track by' expression to specify unique keys. Repeater: arg
> in
> > commandLineArgs, Duplicate key: string:-Xss256k, Duplicate value:
> -Xss256k
> >
>
> This was the clue I needed.
>
> I added this line to the end of solr.in.cmd (I'm doing this testing on
> Windows):
>
> set SOLR_OPTS=%SOLR_OPTS% -Xss256k
>
> With that change and a Solr restart, the Args information disappeared
> from the admin UI.
>
> Somewhere, likely in your include script, you have defined custom
> arguments that have duplicated the -Xss256k and
> -XX:+UseGCLogFileRotation arguments that are included by default.  There
> may be other duplicates, but those are the ones that were included in
> the error information you shared.  If you adjust the startup
> configuration so that there are no duplicate commandline arguments, then
> restart Solr, it should display.
>
> This does mean that Solr has a bug in the admin UI, but it's one that
> you can work around by removing duplicate arguments.  The angular code
> used for the argument display cannot handle duplicate entries.  Here's
> the issue I created for the problem:
>
> https://issues.apache.org/jira/browse/SOLR-11645
>
> There's a patch attached to the issue that fixes the problem for me, and
> some instructions for fixing up a binary download with that change
> rather than a source checkout.
>
> Thanks,
> Shawn
>
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

get all tokens from TokenStream in my custom filter

[kumar gaurav]

Hi

I need to get full field value from TokenStream in my custom filter class .

I am using this

stream.reset();
while (tStream.incrementToken()) {
term += " "+charTermAttr.toString();
}
stream.end();
stream.close();

this is ending streaming . no token is producing if i am using this .

I want to get full string without hampering token creation .

Eric ! Are you there ? :)  Anyone Please help  ?

Solr cloud inquiry

[kasinger, james]

Hello folks,



To start, we have a sharded solr cloud configuration running solr version 5.1.0 
. During shard to shard communication there is a problem state where queries 
are sent to a replica, and on that replica the storage is inaccessible. The 
node is healthy so it’s still taking requests which get piled up waiting to 
read from disk resulting in a latency increase. We’ve tried resolving this 
storage inaccessibility but it appears related to AWS ebs issues.  Has anyone 
encountered the same issue?

thanks

solr expertise

[Niraj Aswani]

Hi,

Apologies if this is not the relevant question for this mailing list but we
are looking for a solr expert and were wondering If this is the right
mailing list to send the job description. If there is any other list,
please can you suggest?

Regards,
Niraj

Re: Error when indexing EML files in Solr 7.1.0

[Zheng Lin Edwin Yeo]

Hi Karthik,

Thanks for the update.

I see from the JIRA that it is still unresolved, meaning we can't index EML
files to Solr 7.1.0 for the time being?

Also, when the patch is ready, are we able to apply the patch to the
current Solr 7.1.0? Or do we have to wait for the next release of Solr?

Regards.
Edwin


On 15 November 2017 at 23:35, Karthik Ramachandran  wrote:

> JIRA already exists, https://issues.apache.org/jira/browse/SOLR-11622.
>
>
> On Mon, Nov 13, 2017 at 5:55 PM, Zheng Lin Edwin Yeo  >
> wrote:
>
> > Hi Erick,
> >
> > I have added the apache-mime4j-core-0.7.2.jar in the Java Build Path of
> the
> > Eclipse, but it is also not working.
> >
> > Regards,
> > Edwin
> >
> > On 13 November 2017 at 23:33, Erick Erickson 
> > wrote:
> >
> > > Where are you getting your mime4j file? MimeConfig is in
> > > /extraction/lib/apache-mime4j-core-0.7.2.jar and you need to make sure
> > > you're including that at a guess.
> > >
> > > Best,
> > > Erick
> > >
> > > On Mon, Nov 13, 2017 at 6:15 AM, Zheng Lin Edwin Yeo
> > >  wrote:
> > > > Hi,
> > > >
> > > > I am using Solr 7.1.0, and I am trying to index EML files using the
> > > > SimplePostTools.
> > > >
> > > > However, I get the following error
> > > >
> > > > java.lang.NoClassDefFoundError:
> > > > org/apache/james/mime4j/stream/MimeConfig$Builder
> > > >
> > > >
> > > > Is there any we class or dependencies which I need to add as compared
> > to
> > > > Solr 6?
> > > >
> > > > The indexing is ok for other files type like .doc, .ppt. I only face
> > the
> > > > error when indexing .eml files.
> > > >
> > > > Regards,
> > > > Edwin
> > >
> >
>

Re: Solr cloud inquiry

[Jaroslaw Rozanski]

Hi,

It is interesting that node reports healthy despite store access issue.
That node should be marked down if it can't open the core backing up
sharded collection.

Maybe if you could share exceptions/errors that you see in console/logs. 

I have experienced issues with replica node not responding in timely
manner due to performance issues but that does not seem to match your
case.


--
Jaroslaw Rozanski 

On Wed, 15 Nov 2017, at 22:49, kasinger, james wrote:
> Hello folks,
> 
> 
> 
> To start, we have a sharded solr cloud configuration running solr version
> 5.1.0 . During shard to shard communication there is a problem state
> where queries are sent to a replica, and on that replica the storage is
> inaccessible. The node is healthy so it’s still taking requests which get
> piled up waiting to read from disk resulting in a latency increase. We’ve
> tried resolving this storage inaccessibility but it appears related to
> AWS ebs issues.  Has anyone encountered the same issue?
> 
> thanks

Re: solr expertise

[Gora Mohanty]

On 16 November 2017 at 03:54, Niraj Aswani  wrote:
>
> Hi,
>
> Apologies if this is not the relevant question for this mailing list but we
> are looking for a solr expert and were wondering If this is the right
> mailing list to send the job description. If there is any other list,
> please can you suggest?

Please see https://wiki.apache.org/solr/Support

Regards,
Gora

RE: How to routing document for send to particular shard range

[Ketan Thanki]

Hi,

Can  someone please let me for my below mentions use-case.

Regards,
Ketan

-Original Message-
From: Ketan Thanki 
Sent: Wednesday, November 15, 2017 3:42 PM
To: 'solr-user@lucene.apache.org'
Subject: RE: How to routing document for send to particular shard range

Thanks Erik,

I have re-mentions it as some required details are missing in my mail. Using 
CloudSolrClient my test case as below.
I have used routing key as projectId/2!documentId
 
1: Detail of Insert Document in SolrIndex Document Size: 919551 Document Batch 
Size for insert: 5000 documents in each thread Total thread to Index: 184 
(thread pool size 5) With Routing: 251835 millisecond taken to index documents 
in index (Data index in 1 shard) Without Routing: 325108 millisecond taken to 
index documents in index (Data index in 4 shard)

2: Detail of search document/Retrieve from SolrIndex Document Size: 919551 
Document Batch Size for retrieve: 1 (per query 1 document id used for 
search) Total thread to Search: 93 (thread pool size 3) Total Queries call: 93 
With Routing: 94562 millisecond taken to search documents from index (Data in 1 
shard) Without Routing: 234242 millisecond taken to search documents From index 
(Data in 4 shard)

Retrieval query with parameter used fq & fl also.
 
So above shows my one model data which belongs to one project (also project is 
consisting of many models).
While refer above use-case my data in single shard which gives improvements 
with routing facility.
but when considering the production environment  If I have 800M documents (it 
may be more) in each of my 4 shard and considering above case so what would be 
better from below.
- to distribute project data in two shard
- to distribute project data in single shard

is there any documents limit for shard? And if my document  grow more than 
above mentions so is it causes any issue?

Please do needful.

Regards,
Ketan







 

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Tuesday, November 14, 2017 9:33 PM
To: solr-user
Subject: Re: How to routing document for send to particular shard range

These numbers aren't very useful. inserting how much data? Querying how much 
data? What kinds of queries? Are you indexing in batches or one document at a 
time? Are you using SolrJ and CloudSolrClient?

94 seconds to do _what_? Execute 1,000 queries? Fetch all the documents from 
the shard? Execute one query?

What is "a huge amount of data"? I've seen 300M documents fit on one shard. 
I've seen people claim 1M documents is "huge".

Details matter. You might review:

https://wiki.apache.org/solr/UsingMailingLists

1:inserting data in 4 shard without document routing  time taken ( in
millisecond)  =325108
Inserting data in 1 shard with document routing time time taken ( in
millisecond)  =251835

2: retrieving data from 4 shard without document routing time taken( in 
millisecond)  = 234242 And retrieving data from 1 shard with document routing 
time taken ( in millisecond)= 94562

Best,
Erick

On Tue, Nov 14, 2017 at 6:50 AM, Ketan Thanki  wrote:
> Thanks Amrit ,
>
> Actually we have huge amount of data so that's why thinking to index data 
> into particular shard accept it's looks difficult but need to achieve the 
> performance using document routing for huge data.
>
> With configuration of  4 shard and 4 replica  is it better to distribute the 
> one project data in multiple shard or in one shard which one is  feasible 
> using document routing because needs the best performance while insertion & 
> retrieval of document. And there would be the multiple projects of client 
> which has huge amount of data.
>
> I also taken the reading with 4 shard and 4 replica where without routing 
> data are distribute among all 4 shard  and with routing its distributes in 1 
> shard because of used 1 bit of shard key like projectId/1!DocumentId.my 
> reading looks as below.
> 1:inserting data in 4 shard without document routing  time taken ( in
> millisecond)  =325108 Inserting data in 1 shard with document routing 
> time time taken ( in millisecond)  =251835
>
> 2: retrieving data from 4 shard without document routing time taken( 
> in millisecond)  = 234242 And retrieving data from 1 shard with 
> document routing time taken ( in millisecond)= 94562
>
> As per above reading getting  performance in local  while data in 1 shard but 
> in production there will be huge data so is it need to distribute in 2 shard 
> or in 1 shard which one is feasible for achieve better performance.
>
>
> Regards,
> Ketan
>
> -Original Message-
> From: Amrit Sarkar [mailto:sarkaramr...@gmail.com]
> Sent: Monday, November 13, 2017 8:52 PM
> To: solr-user@lucene.apache.org
> Subject: Re: How to routing document for send to particular shard 
> range
>
> Surely someone else can chim in;
>
> but when you say: "so regarding to it we need to index the particular
>> client data into particular shard so if its  manageable than we will 
>

Re: get all tokens from TokenStream in my custom filter

[Modassar Ather]

Hi,

Are you getting the string but further tokens are not generated? If this is
the case I think no need to end or close the stream in filter.
Hope this helps.

Best,
Modassar

On Thu, Nov 16, 2017 at 1:20 AM, kumar gaurav  wrote:

> Hi
>
> I need to get full field value from TokenStream in my custom filter class .
>
> I am using this
>
> stream.reset();
> while (tStream.incrementToken()) {
> term += " "+charTermAttr.toString();
> }
> stream.end();
> stream.close();
>
> this is ending streaming . no token is producing if i am using this .
>
> I want to get full string without hampering token creation .
>
> Eric ! Are you there ? :)  Anyone Please help  ?
>

DIH not stop

[Can Ezgi Aydemir]

Hi all,

I configured Solr and Cassandra. Running full data import but not stop. Only 
core load during this process, stop it. Seeing that stop dih, not write 
dataimport.properties.

In dataconfig.xml file, i define simplepropertywriter type and filename. But 
not write it in dataimport.properties file.

How can i solve this problem?

Thx

Regards.
Bu e-posta ve ekindekiler gizli bilgiler içeriyor olabilir ve sadece adreslenen 
kişileri ilgilendirir. Eğer adreslenen kişi siz değilseniz, bu e-postayı 
yaymayınız, dağıtmayınız veya kopyalamayınız. Eğer bu e-posta yanlışlıkla size 
gönderildiyse, lütfen bu e-posta ve ekindeki dosyaları sisteminizden siliniz ve 
göndereni hemen bilgilendiriniz. Ayrıca, bu e-posta ve ekindeki dosyaları virüs 
bulaşması ihtimaline karşı taratınız. İŞLEM GIS® bu e-posta ile taşınabilecek 
herhangi bir virüsün neden olabileceği hasarın sorumluluğunu kabul etmez. Bilgi 
için:b...@islem.com.tr This message may contain confidential information and is 
intended only for recipient name. If you are not the named addressee you should 
not disseminate, distribute or copy this e-mail. Please notify the sender 
immediately if you have received this e-mail by mistake and delete this e-mail 
from your system. Finally, the recipient should check this email and any 
attachments for the presence of viruses. İŞLEM GIS® accepts no liability for 
any damage may be caused by any virus transmitted by this email.” For 
information: b...@islem.com.tr

Re: DIH not stop

[Sujay Bawaskar]

I have experience this problem recently with MySQL and after checking
solr.log found that there was a connection timeout from MySQL.
Please check solr.log for any Cassandra connection errors.

Thanks,
Sujay

On Thu, Nov 16, 2017 at 12:29 PM, Can Ezgi Aydemir 
wrote:

> Hi all,
>
> I configured Solr and Cassandra. Running full data import but not stop.
> Only core load during this process, stop it. Seeing that stop dih, not
> write dataimport.properties.
>
> In dataconfig.xml file, i define simplepropertywriter type and filename.
> But not write it in dataimport.properties file.
>
> How can i solve this problem?
>
> Thx
>
> Regards.
> Bu e-posta ve ekindekiler gizli bilgiler içeriyor olabilir ve sadece
> adreslenen kişileri ilgilendirir. Eğer adreslenen kişi siz değilseniz, bu
> e-postayı yaymayınız, dağıtmayınız veya kopyalamayınız. Eğer bu e-posta
> yanlışlıkla size gönderildiyse, lütfen bu e-posta ve ekindeki dosyaları
> sisteminizden siliniz ve göndereni hemen bilgilendiriniz. Ayrıca, bu
> e-posta ve ekindeki dosyaları virüs bulaşması ihtimaline karşı taratınız.
> İŞLEM GIS® bu e-posta ile taşınabilecek herhangi bir virüsün neden
> olabileceği hasarın sorumluluğunu kabul etmez. Bilgi iç
> in:b...@islem.com.tr This message may contain confidential information
> and is intended only for recipient name. If you are not the named addressee
> you should not disseminate, distribute or copy this e-mail. Please notify
> the sender immediately if you have received this e-mail by mistake and
> delete this e-mail from your system. Finally, the recipient should check
> this email and any attachments for the presence of viruses. İŞLEM GIS®
> accepts no liability for any damage may be caused by any virus transmitted
> by this email.” For information: b...@islem.com.tr
>



-- 
Thanks,
Sujay P Bawaskar
M:+91-77091 53669

RE: DIH not stop

[Can Ezgi Aydemir]

First of all thx for reply.

In solr.log file, do not any error or warning. I am sharing this file in below. 
DIH not stop but I wait 30 mins. I have 5 rows in Cassandra.

SOLR.LOG
---
2017-11-16 07:20:36.094 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___  
_   Welcome to Apache Solr™ version 7.1.0
2017-11-16 07:20:36.095 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| 
___| |_ _   Starting in standalone mode on port 8983
2017-11-16 07:20:36.095 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ 
\ | '_|  Install dir: /opt/solr, Default config dir: 
/opt/solr/server/solr/configsets/_default/conf
2017-11-16 07:20:36.120 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 
|___/\___/_|_|Start time: 2017-11-16T07:20:36.097Z
2017-11-16 07:20:36.122 INFO  (main) [   ] o.a.s.u.StartupLoggingUtils Property 
solr.log.muteconsole given. Muting ConsoleAppender named CONSOLE
2017-11-16 07:20:36.147 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using 
system property solr.solr.home: /var/solr/data
2017-11-16 07:20:36.157 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
container configuration from /var/solr/data/solr.xml
2017-11-16 07:20:37.023 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Found 
4 core definitions underneath /var/solr/data
2017-11-16 07:20:37.026 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Cores 
are: [1, Audit, cea2, deneme123]
2017-11-16 07:20:37.034 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
2017-11-16 07:20:37.100 INFO  (main) [   ] o.e.j.s.Server Started @2440ms
2017-11-16 07:20:37.165 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.c.SolrResourceLoader [1] Added 58 libs to classloader, from paths: 
[/opt/solr/contrib/clustering/lib, 
/opt/solr/contrib/dataimporthandler-extras/lib, 
/opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, 
/opt/solr/contrib/velocity/lib, /opt/solr/dist]
2017-11-16 07:20:37.165 INFO  (coreLoadExecutor-6-thread-3) [   x:cea2] 
o.a.s.c.SolrResourceLoader [cea2] Added 58 libs to classloader, from paths: 
[/opt/solr/contrib/clustering/lib, 
/opt/solr/contrib/dataimporthandler-extras/lib, 
/opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, 
/opt/solr/contrib/velocity/lib, /opt/solr/dist]
2017-11-16 07:20:37.165 INFO  (coreLoadExecutor-6-thread-2) [   x:Audit] 
o.a.s.c.SolrResourceLoader [Audit] Added 58 libs to classloader, from paths: 
[/opt/solr/contrib/clustering/lib, 
/opt/solr/contrib/dataimporthandler-extras/lib, 
/opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib, 
/opt/solr/contrib/velocity/lib, /opt/solr/dist]
2017-11-16 07:20:37.421 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
2017-11-16 07:20:37.434 INFO  (coreLoadExecutor-6-thread-3) [   x:cea2] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
2017-11-16 07:20:37.434 INFO  (coreLoadExecutor-6-thread-2) [   x:Audit] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
2017-11-16 07:20:37.859 INFO  (coreLoadExecutor-6-thread-2) [   x:Audit] 
o.a.s.s.IndexSchema [Audit] Schema name=default-config
2017-11-16 07:20:37.865 INFO  (coreLoadExecutor-6-thread-3) [   x:cea2] 
o.a.s.s.IndexSchema [cea2] Schema name=default-config
2017-11-16 07:20:37.866 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.s.IndexSchema [1] Schema name=default-config
2017-11-16 07:20:38.899 INFO  (coreLoadExecutor-6-thread-3) [   x:cea2] 
o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
2017-11-16 07:20:38.909 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
2017-11-16 07:20:38.917 INFO  (coreLoadExecutor-6-thread-2) [   x:Audit] 
o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
2017-11-16 07:20:38.950 INFO  (coreLoadExecutor-6-thread-3) [   x:cea2] 
o.a.s.c.CoreContainer Creating SolrCore 'cea2' using configuration from 
instancedir /var/solr/data/cea2, trusted=true
2017-11-16 07:20:38.962 INFO  (coreLoadExecutor-6-thread-2) [   x:Audit] 
o.a.s.c.CoreContainer Creating SolrCore 'Audit' using configuration from 
instancedir /var/solr/data/Audit, trusted=true
2017-11-16 07:20:38.963 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.c.CoreContainer Creating SolrCore '1' using configuration from 
instancedir /var/solr/data/1, trusted=true
2017-11-16 07:20:39.029 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
2017-11-16 07:20:39.029 INFO  (coreLoadExecutor-6-thread-2) [   x:Audit] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
2017-11-16 07:20:39.029 INFO  (coreLoadExecutor-6-thread-3) [   x:cea2] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
2017-11-16 07:20:39.040 INFO  (coreLoadExecutor-6-thread-1) [   x:1] 
o.a.s.c.SolrCore [[1] ] Opening new SolrCore at [/var/solr/data/1], 
dataDir=[/var/solr/data/1/d