Re: Re: Tagging and excluding Filters with BlockJoin Queries and BlockJoin Faceting

[Mikhail Khludnev]

Sure. There are might mismatch with expectation. However, the first guess
is to put {!tag into beginning. eg, check with
fq={!tag=myTag}{!parent which='isparent:true'}color:blue

On Tue, Aug 23, 2016 at 4:05 PM, Tobias Lorenz  wrote:

> Hi Mikhail,
>
> Thanks for replying so quickly with a suggestion.
>
> I'm a colleague of Stefan and working with him on our project.
>
> We tried composing our solr query with exclusion instructions, and the
> result was that the facet excluded by tag did not show up anymore in the
> result, instead of showing all values.
>
> Your example from the last comment, completed by our exlusion instruction:
>
> json.facet={
>   filter_by_children: {
> type: query,
> q: "isparent:false",
> domain: {
>   blockChildren: "isparent:true"
> },
> facet: {
>   colors: {
> type: terms,
> field: color,
> domain:{
>   excludeTags:myTag
> },
> facet: {
>   productsCount: "unique(_root_)"
> }
>   }
> }
>   }
> }
>
>
> and the corresponding filter query:
>
> fq={!parent which='isparent:true'}{!tag=myTag}color:blue
>
>
> Either this feature is not working yet, or we are making a mistake using
> it.
> Of course we know it's still in development right now.
>
> Might you please have a look if we are doing something obviously wrong?
>
> Thanks,
> Tobias
>
>
>
> >The last comment at https://issues.apache.org/jira/browse/SOLR-8998 shows
> >the current verbose json.facet syntax which provides aggregated facet
> >counts already. It's a little bit slower that child.facet.field.
> >Nevertheless, you can take this sample and add exclusion instructions
> into.
> >It should work. Let me know how does it, please.
> >
> >On Wed, Aug 17, 2016 at 5:35 PM, Stefan Moises 
> wrote:
> >
> >> Hi Mikhail,
> >>
> >> thanks for the info ... what is the advantage of using the JSON FACET
> API
> >> compared to the standard BlockJoinQuery features?
> >>
> >> Is there already anybody working on the tagging/exclusion feature or is
> >> there any timeframe for it? There wasn't any discussion yet in SOLR-8998
> >> about exclusions, was there?
> >>
> >> Thank you very much,
> >>
> >> best,
> >>
> >> Stefan
> >>
> >>
> >> Am 17.08.16 um 15:26 schrieb Mikhail Khludnev:
> >>
> >> Stefan,
> >>> child.facet.field never intend to support exclusions. My preference is
> to
> >>> implement it under json.facet that's discussed under
> >>> https://issues.apache.org/jira/browse/SOLR-8998.
> >>>
> >>> On Wed, Aug 17, 2016 at 3:52 PM, Stefan Moises 
> >>> wrote:
> >>>
> >>> Hey girls and guys,
> 
>  for a long time we have been using our own BlockJoin Implementation,
>  because for our Shop Systems a lot of requirements that we had were
> not
>  implemented in solr.
> 
>  As we now had a deeper look into how far the standard has come, we saw
>  that BlockJoin and faceting on children is now part of the standard,
>  which
>  is pretty cool.
>  When I tried to refactor our external code to use that now, I stumbled
>  upon one non-working feature with BlockJoins that still keeps us from
>  using
>  it:
> 
>  It seems that tagging and excluding Filters with BlockJoin Faceting
>  simply
>  does not work yet.
> 
>  Simple query:
> 
>  &qt=products
>  &q={!parent which='isparent:true'}shirt AND isparent:false
>  &facet=true
>  &fq={!parent which='isparent:true'}{!tag=myTag}color:grey
>  &child.facet.field={!ex=myTag}color
> 
> 
>  Gives us:
>  o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
>  undefined field: "{!ex=myTag}color"
>   at org.apache.solr.schema.IndexSchema.getField(IndexSchema.
>  java:1231)
> 
> 
>  Does somebody have an idea?
> 
> 
>  Best,
>  Stefan
> 
>  --
>  --
>  
>  Stefan Moises
>  Manager Research & Development
>  shoptimax GmbH
>  Ulmenstraße 52 H
>  90443 Nürnberg
>  Tel.: 0911/25566-0
>  Fax: 0911/25566-29
>  moi...@shoptimax.de
>  http://www.shoptimax.de
> 
>  Geschäftsführung: Friedrich Schreieck
>  Ust.-IdNr.: DE 814340642
>  Amtsgericht Nürnberg HRB 21703
> 
> 
> 
> 
> >>>
> >> --
> >> --
> >> 
> >> Stefan Moises
> >> Manager Research & Development
> >> shoptimax GmbH
> >> Ulmenstraße 52 H
> >> 90443 Nürnberg
> >> Tel.: 0911/25566-0
> >> Fax: 0911/25566-29
> >> moi...@shoptimax.de
> >> http://www.shoptimax.de
> >>
> >> Geschäftsführung: Friedrich Schreieck
> >> Ust.-IdNr.: DE 814340642
> >> Amtsgericht Nürnberg HRB 21703
> >>   
> >>
> >>
> >
> >
> >--
> >Sincerely yours
> >Mikhail Khludnev
> >
> >
>



-- 
Sincerely yours
Mikhail Khludnev

AW: Re: Tagging and excluding Filters with BlockJoin Queries and BlockJoin Faceting

[Tobias Lorenz]

I tried that too, with no effect.

The excluded facet just disappears completely (even the value that is filtered 
on in the fq) when using the exclusion that has been tagged, like it did before.
When using a random exclusion (e.g. foo) that facet is visible again in the 
result set, but that's obviously not helpful, I just tried to see what it would 
do.

So this is my current research result:

When excluding a facet which has been tagged in a filter query, this facet 
corresponding to the fq's tag disappears in the result set in solr 6.1 when 
using BlockJoin Queries and json facets (which it shouldn't).

Let me know if you want me to do more research or have one more idea.


-Ursprüngliche Nachricht-
Von: Mikhail Khludnev [mailto:m...@apache.org] 
Gesendet: Mittwoch, 24. August 2016 09:06
An: solr-user 
Betreff: Re: Re: Tagging and excluding Filters with BlockJoin Queries and 
BlockJoin Faceting

Sure. There are might mismatch with expectation. However, the first guess is to 
put {!tag into beginning. eg, check with fq={!tag=myTag}{!parent 
which='isparent:true'}color:blue

On Tue, Aug 23, 2016 at 4:05 PM, Tobias Lorenz  wrote:

> Hi Mikhail,
>
> Thanks for replying so quickly with a suggestion.
>
> I'm a colleague of Stefan and working with him on our project.
>
> We tried composing our solr query with exclusion instructions, and the 
> result was that the facet excluded by tag did not show up anymore in 
> the result, instead of showing all values.
>
> Your example from the last comment, completed by our exlusion instruction:
>
> json.facet={
>   filter_by_children: {
> type: query,
> q: "isparent:false",
> domain: {
>   blockChildren: "isparent:true"
> },
> facet: {
>   colors: {
> type: terms,
> field: color,
> domain:{
>   excludeTags:myTag
> },
> facet: {
>   productsCount: "unique(_root_)"
> }
>   }
> }
>   }
> }
>
>
> and the corresponding filter query:
>
> fq={!parent which='isparent:true'}{!tag=myTag}color:blue
>
>
> Either this feature is not working yet, or we are making a mistake 
> using it.
> Of course we know it's still in development right now.
>
> Might you please have a look if we are doing something obviously wrong?
>
> Thanks,
> Tobias
>
>
>
> >The last comment at https://issues.apache.org/jira/browse/SOLR-8998 
> >shows the current verbose json.facet syntax which provides aggregated 
> >facet counts already. It's a little bit slower that child.facet.field.
> >Nevertheless, you can take this sample and add exclusion instructions
> into.
> >It should work. Let me know how does it, please.
> >
> >On Wed, Aug 17, 2016 at 5:35 PM, Stefan Moises 
> wrote:
> >
> >> Hi Mikhail,
> >>
> >> thanks for the info ... what is the advantage of using the JSON 
> >> FACET
> API
> >> compared to the standard BlockJoinQuery features?
> >>
> >> Is there already anybody working on the tagging/exclusion feature 
> >> or is there any timeframe for it? There wasn't any discussion yet 
> >> in SOLR-8998 about exclusions, was there?
> >>
> >> Thank you very much,
> >>
> >> best,
> >>
> >> Stefan
> >>
> >>
> >> Am 17.08.16 um 15:26 schrieb Mikhail Khludnev:
> >>
> >> Stefan,
> >>> child.facet.field never intend to support exclusions. My 
> >>> preference is
> to
> >>> implement it under json.facet that's discussed under 
> >>> https://issues.apache.org/jira/browse/SOLR-8998.
> >>>
> >>> On Wed, Aug 17, 2016 at 3:52 PM, Stefan Moises 
> >>> 
> >>> wrote:
> >>>
> >>> Hey girls and guys,
> 
>  for a long time we have been using our own BlockJoin 
>  Implementation, because for our Shop Systems a lot of 
>  requirements that we had were
> not
>  implemented in solr.
> 
>  As we now had a deeper look into how far the standard has come, 
>  we saw that BlockJoin and faceting on children is now part of the 
>  standard, which is pretty cool.
>  When I tried to refactor our external code to use that now, I 
>  stumbled upon one non-working feature with BlockJoins that still 
>  keeps us from using
>  it:
> 
>  It seems that tagging and excluding Filters with BlockJoin 
>  Faceting simply does not work yet.
> 
>  Simple query:
> 
>  &qt=products
>  &q={!parent which='isparent:true'}shirt AND isparent:false 
>  &facet=true &fq={!parent 
>  which='isparent:true'}{!tag=myTag}color:grey
>  &child.facet.field={!ex=myTag}color
> 
> 
>  Gives us:
>  o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException:
>  undefined field: "{!ex=myTag}color"
>   at org.apache.solr.schema.IndexSchema.getField(IndexSchema.
>  java:1231)
> 
> 
>  Does somebody have an idea?
> 
> 
>  Best,
>  Stefan
> 
>  --
>  --
>  
>  Stefan Moises
>  Manager Research & Development
>  shoptimax GmbH
>  Ulmenstraße 52 H

Re: Custom handler/content stream loader

[Jamie Johnson]

I did start start with this but it's a limited approach since it only works
with text fields.  Right now I'm using this with a bunch of custom fields
extended to support payloads but that is expensive to maintain between
versions, especially when APIs change so I'm looking for a less invasive
way of supporting the same capability. I believe I have a Lucene solution
for handling any field type, though i will obviously need to test it, if I
can figure out the best way to do the custom building of the document and
how to marshal it to the server and from the client.

On Aug 23, 2016 11:05 PM, "Alexandre Rafalovitch" 
wrote:

> Have you tried starting with the DelimitedPayloadTokenFilterFactory?
> There is a sample configuration in the shipped examples:
> https://github.com/apache/lucene-solr/blob/releases/
> lucene-solr/6.1.0/solr/example/example-DIH/solr/db/
> conf/managed-schema#L625
>
> Regards,
> Alex.
> 
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 24 August 2016 at 04:22, Jamie Johnson  wrote:
> > I have a need to build custom field types that store additional metadata
> at
> > the field level in a payload.  I was thinking that I could satisfy this
> by
> > building a custom UpdateRequest that captured this additional information
> > in XML, but I am not really sure how to get at this additional
> information
> > on the server side.  Would I need to implement a custom RequestHandler to
> > handle the update, could I add a custom ContentStreamLoader to parse the
> > XML, how do I customize the creation of the lucene document once I have
> the
> > XML?  Any help/direction would really be appreciated.
> >
> > -Jamie
>

Re: Is it safe to upgrade an existing field to docvalues?

[Toke Eskildsen]

On Tue, 2016-08-23 at 20:01 +, Ronald Wood wrote:
> In general, is there a way to migrate existing indexes (we have
> petabytes of data) by enabling docvalues and incrementally re-
> indexing? We expect the latter would take a month using an atomic
> update process.

I did write a tool for that at some point, that works with Solr 4.x.
https://github.com/netarchivesuite/dvenabler

It is not polished though, and the main problem is that it is slow to
convert. Some details at https://sbdevel.wordpress.com/2014/12/15/chang
ing-field-type-in-lucenesolr/

- Toke Eskildsen

Re: Is it safe to upgrade an existing field to docvalues?

[Alessandro Benedetti]

Hi Toke !
Good stuff !

So basically using your tool you build a copy of the index ( similar to
what optimize does) without affecting the main index right ?
So the procedure would be :

1) Solr is running
2) Run the tool pointing to the Solr Index, this will be slow but will
generate a new index copying from the stored content to docValues
3) Stop Solr
4) Change Solr schema enabling docValues
5) Point to the new converted index
6) Start Solr

Am I right ?
This is actually a useful tool when re-indexing could be extremely long.

Cheers

On Wed, Aug 24, 2016 at 12:05 PM, Toke Eskildsen 
wrote:

> On Tue, 2016-08-23 at 20:01 +, Ronald Wood wrote:
> > In general, is there a way to migrate existing indexes (we have
> > petabytes of data) by enabling docvalues and incrementally re-
> > indexing? We expect the latter would take a month using an atomic
> > update process.
>
> I did write a tool for that at some point, that works with Solr 4.x.
> https://github.com/netarchivesuite/dvenabler
>
> It is not polished though, and the main problem is that it is slow to
> convert. Some details at https://sbdevel.wordpress.com/2014/12/15/chang
> ing-field-type-in-lucenesolr/
>
> - Toke Eskildsen
>
>


-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

equivalent of localhost_access_log for solr 5

[elisabeth benoit]

Hello,

I'd like to know what is the best way to have the equivalent of
tomcat localhost_access_log  for solr 5?

Best regards,
Elisabeth

Re: Settings for DocValues

[Alessandro Benedetti]

In addition to what Erick said, there is actually a smaller implication
between Stored and docValues as well .
With latest Solr releases docValues data structures are used to fetch the
stored content.
In the case of multi-valued fields, we can have inconsistency with the
ordering of the multiple values.

So, if the case applies, I suggest to specify the additional attribute.

In the past, Solr guaranteed that retrieval of multi-valued fields would
> preserve the order of values. Because values may now be retrieved from
> column-stored fields (docValues="true"), in conjunction with the fact
> that DocValues
>  do not
> currently preserve order, means that users should set
> useDocValuesAsStored="false"
>  to
> prevent future optimizations from using the column-stored values over the
> row-stored values when fields have both stored="true" and docValues="true"


Cheers

On Wed, Aug 24, 2016 at 4:42 AM, Zheng Lin Edwin Yeo 
wrote:

> Hi Erick,
>
> Thanks for the explanation.
>
> Regards,
> Edwin
>
>
> On 24 August 2016 at 02:27, Erick Erickson 
> wrote:
>
> > These are orthogonal. The indexed and docvalues
> > structures are very different. One is useful for
> > searching and the other is useful for faceting and the like.
> >
> > If you set indexed="false" and docValues="true", then try
> > to search on the field you roughly do a "table scan" which
> > is very slow.
> >
> > Rule of thumb:
> > If you're searching set indexed="true".
> > If you're faceting/grouping/sorting set docValues="true".
> >
> > Best,
> > Erick
> >
> > On Tue, Aug 23, 2016 at 8:14 AM, Zheng Lin Edwin Yeo
> >  wrote:
> > > Hi,
> > >
> > > Would like to find out, if we set docValues="true" in my configuration
> of
> > > the fieldType in schema,xml, must we set the corresponding
> > indexed="false"
> > > and stored="false"
> > >
> > > Will there be any implication if I set my indexed="true and
> > stored="true"?
> > >
> > > I'm using Solr 6.1.0
> > >
> > > Regards,
> > > Edwin
> >
>



-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

another log question about solr 5

[elisabeth benoit]

Hello again,

We're planning on using solr 5.5.2 on production, using installation
script install_solr_service.sh.

I was wondering was is the right way to prevent solr 5 from creating a new
log file at every startup  (and renaming the actual file mv
"$SOLR_LOGS_DIR/solr_gc.log" "$SOLR_LOGS_DIR/solr_gc_log_$(date
+"%Y%m%d_%H%M")"

Thanks,
Elisabeth

Re: Is it safe to upgrade an existing field to docvalues?

[Ronald Wood]

Thanks Toke. I’ve read some of your other helpful blog entries but I missed 
that one. 

Did you find you had to do a full conversion all at once because simply turning 
on docvalues in the schema caused issues? 

I ask because my presupposition has been that we could turn it on without any 
harm as we incrementally converted our indexes.

I have actually written a tool that will round robin between different indexes 
on the same server and update 10,000 items every two minutes by doing a simple 
atomic update (updating a date field). This actually works well in my testing. 

But this won’t work if enabling docvalues in the schema will lead to errors 
when fields don’t have docvalues actually populated. I.e.. the 
“IllegalStateException: unexpected docvalues type NONE for field 'id' 
(expected=SORTED)” error I see.

I’m still trying to get to the bottom of whether that error means I cannot 
safely do an incremental conversion in-place.

Your approach may come in handy if we cannot.

-Ronald.

On 8/24/16, 07:42, "Alessandro Benedetti"  wrote:

Hi Toke !
Good stuff !

So basically using your tool you build a copy of the index ( similar to
what optimize does) without affecting the main index right ?
So the procedure would be :

1) Solr is running
2) Run the tool pointing to the Solr Index, this will be slow but will
generate a new index copying from the stored content to docValues
3) Stop Solr
4) Change Solr schema enabling docValues
5) Point to the new converted index
6) Start Solr

Am I right ?
This is actually a useful tool when re-indexing could be extremely long.

Cheers

On Wed, Aug 24, 2016 at 12:05 PM, Toke Eskildsen 
wrote:

> On Tue, 2016-08-23 at 20:01 +, Ronald Wood wrote:
> > In general, is there a way to migrate existing indexes (we have
> > petabytes of data) by enabling docvalues and incrementally re-
> > indexing? We expect the latter would take a month using an atomic
> > update process.
>
> I did write a tool for that at some point, that works with Solr 4.x.
> https://github.com/netarchivesuite/dvenabler
>
> It is not polished though, and the main problem is that it is slow to
> convert. Some details at https://sbdevel.wordpress.com/2014/12/15/chang
> ing-field-type-in-lucenesolr/
>
> - Toke Eskildsen
>
>


-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Re: Is it safe to upgrade an existing field to docvalues?

[Shawn Heisey]

On 8/23/2016 2:01 PM, Ronald Wood wrote:
> In general, is there a way to migrate existing indexes (we have petabytes of 
> data) by enabling docvalues and incrementally re-indexing? We expect the 
> latter would take a month using an atomic update process.

One way to handle it is to build a new index with an updated
configuration, then switch to the new index.  Since you're not running
SolrCloud, you can switch by swapping the cores.  If you were running
SolrCloud, you'd need to alias the old name to the new collection, which
might involve deleting the old collection first.  Swapping cores in
cloud mode will break things.

The other replies you've gotten are interesting.  The approach using
Atomic Updates will only work if your index meets the requirements for
Atomic Updates.

https://wiki.apache.org/solr/Atomic_Updates#Caveats_and_Limitations

You've already said it would take a month using atomic update ... which
might mean you've already thought about whether or not your index meets
the requirements.

Toke's tool looks quite interesting, and would probably do the job a lot
faster than any other method.

Thanks,
Shawn

Re: Is it safe to upgrade an existing field to docvalues?

[Ronald Wood]

Yes, Shawn, our indexes meet the requirements for atomic updates. 

We actually depend on atomic updates since our users can alter metadata about 
any of our indexed records. We don’t have to incur the cost of a full re-index 
of a record for every change. This is especially critical when a user does a 
bulk update of the status of 1 million records. ☺

- Ronald S. Wood


On 8/24/16, 09:56, "Shawn Heisey"  wrote:

On 8/23/2016 2:01 PM, Ronald Wood wrote:
> In general, is there a way to migrate existing indexes (we have petabytes 
of data) by enabling docvalues and incrementally re-indexing? We expect the 
latter would take a month using an atomic update process.

One way to handle it is to build a new index with an updated
configuration, then switch to the new index.  Since you're not running
SolrCloud, you can switch by swapping the cores.  If you were running
SolrCloud, you'd need to alias the old name to the new collection, which
might involve deleting the old collection first.  Swapping cores in
cloud mode will break things.

The other replies you've gotten are interesting.  The approach using
Atomic Updates will only work if your index meets the requirements for
Atomic Updates.

https://wiki.apache.org/solr/Atomic_Updates#Caveats_and_Limitations

You've already said it would take a month using atomic update ... which
might mean you've already thought about whether or not your index meets
the requirements.

Toke's tool looks quite interesting, and would probably do the job a lot
faster than any other method.

Thanks,
Shawn

Re: Is it safe to upgrade an existing field to docvalues?

[Alessandro Benedetti]

I am sorry Ronald but :
"  ask because my presupposition has been that we could turn it on without
any harm as we incrementally converted our indexes."

This is not possible, if you change the schema and then slowly update the
documents you are introducing inconsistency that will reflect in sorting
and faceting.
Because solr will check the field attributes, will see docValues, but then
will find only partial docValues.
So the docValue for some documents will be null.

You need to go live one-shot.
This is the reason Shawn and Toke suggest a parallel index, with the
docValues enabled and finally you swap.

Cheers

On Wed, Aug 24, 2016 at 2:56 PM, Shawn Heisey  wrote:

> On 8/23/2016 2:01 PM, Ronald Wood wrote:
> > In general, is there a way to migrate existing indexes (we have
> petabytes of data) by enabling docvalues and incrementally re-indexing? We
> expect the latter would take a month using an atomic update process.
>
> One way to handle it is to build a new index with an updated
> configuration, then switch to the new index.  Since you're not running
> SolrCloud, you can switch by swapping the cores.  If you were running
> SolrCloud, you'd need to alias the old name to the new collection, which
> might involve deleting the old collection first.  Swapping cores in
> cloud mode will break things.
>
> The other replies you've gotten are interesting.  The approach using
> Atomic Updates will only work if your index meets the requirements for
> Atomic Updates.
>
> https://wiki.apache.org/solr/Atomic_Updates#Caveats_and_Limitations
>
> You've already said it would take a month using atomic update ... which
> might mean you've already thought about whether or not your index meets
> the requirements.
>
> Toke's tool looks quite interesting, and would probably do the job a lot
> faster than any other method.
>
> Thanks,
> Shawn
>
>


-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Three Lucene hackdays coming soon

[Charlie Hull]

Hi all,

We're participating in three events in Glasgow, London and Boston over 
the next couple of months - details in my blog at 
http://www.flax.co.uk/blog/2016/08/24/not-one-three-lucene-hackdays-coming-soon/ 
. Very interested in any ideas the community has about what we should 
try to achieve!


Cheers

Charlie

--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Re: equivalent of localhost_access_log for solr 5

[Shawn Heisey]

On 8/24/2016 5:44 AM, elisabeth benoit wrote:
> I'd like to know what is the best way to have the equivalent of tomcat
> localhost_access_log for solr 5?

I don't know for sure what that is, but it sounds like a request log. 
If you edit server/etc/jetty.xml you will find a commented out section
of configuration that enables a request log.  The header says "Configure
Request Log".  If that's what you want, just uncomment that section and
restart Solr.

Thanks,
Shawn

ask question about solr 1.6 or 6

[Hamed Gh]

hello,dear
 i have a question about solr 6 or 1.6

question: how do i search or query in solr 1.6 that bring my filed values
without seperate words although space between iranian-persian or arabic
language character
for example:
../tableName/select?q=*:*&fq={!q.op=AND df=title}دانش آموزان
گاج&rows=0&facet.pivot=title&facet=true&facet.pivot.mincount=1

query response:

021*:*title{!q.op=AND df=title}دانش
آموزان گاج0true1

titleآموزان75529titleدانش75529titleگاج75529

---

i want دانش آموزان گاج just.
i wait for  your answer.
thank you

Re: equivalent of localhost_access_log for solr 5

[elisabeth benoit]

Thanks a lot for your answer.

Best regards,
elisabeth

2016-08-24 16:16 GMT+02:00 Shawn Heisey :

> On 8/24/2016 5:44 AM, elisabeth benoit wrote:
> > I'd like to know what is the best way to have the equivalent of tomcat
> > localhost_access_log for solr 5?
>
> I don't know for sure what that is, but it sounds like a request log.
> If you edit server/etc/jetty.xml you will find a commented out section
> of configuration that enables a request log.  The header says "Configure
> Request Log".  If that's what you want, just uncomment that section and
> restart Solr.
>
> Thanks,
> Shawn
>
>

Re: Is it safe to upgrade an existing field to docvalues?

[Ronald Wood]

OK. Thank you, Alessandro, for clarifying this matter.

The reason I wasn’t sure about this is that this is somewhat ambiguous in the 
documentation. In the 6.1 Guide I see: “If you have already indexed data into 
your Solr index, you will need to completely re-index your content after 
changing your field definitions in schema.xml in order to successfully use 
docValues.”  Maybe that should read “...in order to successfully sort or facet 
on that field or use any other features that depend on docValues. Partially 
converted indexes will result in exceptions because of inconsistent data for 
docValues. ”

Moreover, as I mentioned in my first post, I saw some indication that Solr will 
fall back to using the UninvertingReader if it doesn’t find docValues as 
expected.

In my testing, I did see that /export was definitely an all or nothing case: 
all data had to be docValues before I could get data. /select mostly works – 
except when it occasionally doesn’t.

---

*I wonder if I can make a proposal*: would it be possible to add a property to 
the schema called useDocValues=true/false, defaulting to true?

The idea would be that if docValues=true, indexing docValues would be as 
before, but Solr would not use them as long as useDocValues=false.

Once anyone using this is sure that docValues are fully indexed, set 
useDocValues=true (or remove), and Solr would behave as now.

I spent a little time going down into the code and at first glance this seems 
feasible. I would be willing to log the ticket and perhaps provide a patch.

Does this sound feasible to anyone else? I am uncertain if this requires any 
changes at the Lucene level, but looking at Solr core code all the switching is 
done in Solr on field.hasDocValues. The code would be amended to 
(field.hasDocValues && field.useDocValues) throughout.

I would have to imagine this would be helpful to others out there with large 
amounts of data to migrate.

- Ronald S. Wood 


On 8/24/16, 10:14, "Alessandro Benedetti"  wrote:

I am sorry Ronald but :
"  ask because my presupposition has been that we could turn it on without
any harm as we incrementally converted our indexes."

This is not possible, if you change the schema and then slowly update the
documents you are introducing inconsistency that will reflect in sorting
and faceting.
Because solr will check the field attributes, will see docValues, but then
will find only partial docValues.
So the docValue for some documents will be null.

You need to go live one-shot.
This is the reason Shawn and Toke suggest a parallel index, with the
docValues enabled and finally you swap.

Cheers

On Wed, Aug 24, 2016 at 2:56 PM, Shawn Heisey  wrote:

> On 8/23/2016 2:01 PM, Ronald Wood wrote:
> > In general, is there a way to migrate existing indexes (we have
> petabytes of data) by enabling docvalues and incrementally re-indexing? We
> expect the latter would take a month using an atomic update process.
>
> One way to handle it is to build a new index with an updated
> configuration, then switch to the new index.  Since you're not running
> SolrCloud, you can switch by swapping the cores.  If you were running
> SolrCloud, you'd need to alias the old name to the new collection, which
> might involve deleting the old collection first.  Swapping cores in
> cloud mode will break things.
>
> The other replies you've gotten are interesting.  The approach using
> Atomic Updates will only work if your index meets the requirements for
> Atomic Updates.
>
> https://wiki.apache.org/solr/Atomic_Updates#Caveats_and_Limitations
>
> You've already said it would take a month using atomic update ... which
> might mean you've already thought about whether or not your index meets
> the requirements.
>
> Toke's tool looks quite interesting, and would probably do the job a lot
> faster than any other method.
>
> Thanks,
> Shawn
>
>


-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

restore failure on Solr 6.0.0

[Siddhartha Singh Sandhu]

Hi,

The API to backup(action=BACKUP) has not been implemented for Solr 6.0.0. I
used the replication end pt. to create a backup which it did. When I try to
restore from backup Solr throughs:

"exception":"java.lang.OutOfMemoryError: Java heap space"

 I would believe that the amount of memory used to create the backup should
be enough to restore it. Why do I have to give it more JVM?

Regards,

Sid.

Re: Is it safe to upgrade an existing field to docvalues?

[Ronald Wood]

I created https://issues.apache.org/jira/browse/SOLR-9437 for the proposal 
below.

I suppose beside feasibility, there’s the question of whether the change is 
needed by others. I’d love to hear if it meets anyone else’s needs.

- Ronald S. Wood 


On 8/24/16, 15:08, "Ronald Wood"  wrote:

OK. Thank you, Alessandro, for clarifying this matter.

The reason I wasn’t sure about this is that this is somewhat ambiguous in 
the documentation. In the 6.1 Guide I see: “If you have already indexed data 
into your Solr index, you will need to completely re-index your content after 
changing your field definitions in schema.xml in order to successfully use 
docValues.”  Maybe that should read “...in order to successfully sort or facet 
on that field or use any other features that depend on docValues. Partially 
converted indexes will result in exceptions because of inconsistent data for 
docValues. ”

Moreover, as I mentioned in my first post, I saw some indication that Solr 
will fall back to using the UninvertingReader if it doesn’t find docValues as 
expected.

In my testing, I did see that /export was definitely an all or nothing 
case: all data had to be docValues before I could get data. /select mostly 
works – except when it occasionally doesn’t.

---

*I wonder if I can make a proposal*: would it be possible to add a property 
to the schema called useDocValues=true/false, defaulting to true?

The idea would be that if docValues=true, indexing docValues would be as 
before, but Solr would not use them as long as useDocValues=false.

Once anyone using this is sure that docValues are fully indexed, set 
useDocValues=true (or remove), and Solr would behave as now.

I spent a little time going down into the code and at first glance this 
seems feasible. I would be willing to log the ticket and perhaps provide a 
patch.

Does this sound feasible to anyone else? I am uncertain if this requires 
any changes at the Lucene level, but looking at Solr core code all the 
switching is done in Solr on field.hasDocValues. The code would be amended to 
(field.hasDocValues && field.useDocValues) throughout.

I would have to imagine this would be helpful to others out there with 
large amounts of data to migrate.

- Ronald S. Wood 


On 8/24/16, 10:14, "Alessandro Benedetti"  wrote:

I am sorry Ronald but :
"  ask because my presupposition has been that we could turn it on 
without
any harm as we incrementally converted our indexes."

This is not possible, if you change the schema and then slowly update 
the
documents you are introducing inconsistency that will reflect in sorting
and faceting.
Because solr will check the field attributes, will see docValues, but 
then
will find only partial docValues.
So the docValue for some documents will be null.

You need to go live one-shot.
This is the reason Shawn and Toke suggest a parallel index, with the
docValues enabled and finally you swap.

Cheers

On Wed, Aug 24, 2016 at 2:56 PM, Shawn Heisey  
wrote:

> On 8/23/2016 2:01 PM, Ronald Wood wrote:
> > In general, is there a way to migrate existing indexes (we have
> petabytes of data) by enabling docvalues and incrementally 
re-indexing? We
> expect the latter would take a month using an atomic update process.
>
> One way to handle it is to build a new index with an updated
> configuration, then switch to the new index.  Since you're not running
> SolrCloud, you can switch by swapping the cores.  If you were running
> SolrCloud, you'd need to alias the old name to the new collection, 
which
> might involve deleting the old collection first.  Swapping cores in
> cloud mode will break things.
>
> The other replies you've gotten are interesting.  The approach using
> Atomic Updates will only work if your index meets the requirements for
> Atomic Updates.
>
> https://wiki.apache.org/solr/Atomic_Updates#Caveats_and_Limitations
>
> You've already said it would take a month using atomic update ... 
which
> might mean you've already thought about whether or not your index 
meets
> the requirements.
>
> Toke's tool looks quite interesting, and would probably do the job a 
lot
> faster than any other method.
>
> Thanks,
> Shawn
>
>


-- 
--

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symm

Solr 6.1 Won't Start on Non-bash-Enabled System

[Scott Vanderbilt]

I'm attempting to upgrade a Solr 4.8.1 instance to 6.1. The host OS is 
OpenBSD 6.0.


Attempting to start Solr (as in the on-line QuickStart) with:

  bin/solr start -e cloud -noprompt

fails with the error "env: bash: No such file or directory". Apparently 
bin/solr is a bash script. The OpenBSD host I'm using doesn't have bash 
installed (and unfortunately, no, I can't install it just for this). The 
default shell is ksh.


When I try to use the script I previously used to start the 4.8.1 
version, I get the warnings:


   WARNING: System properties and/or JVM args set.  Consider using
 --dry-run or --exec
   WARNING: Nothing to start, exiting ..."

The old startup script looks like this:

   JAVA_HOME=/usr/local/jdk-1.8.0 JAVA=java JETTY_HOME=/var/solr6
 JETTY_PORT=8983 JETTY_LOGS=/var/solr6/logs
 /usr/local/jdk-1.8.0/bin/java -Dsolr.solr.home=/var/solr6/solr
 -Djava.util.logging.config.file=etc/logging.properties
 -DSTOP.PORT=8079 -DSTOP.KEY=abc123 -jar /var/solr6/start.jar
 --daemon &

If I had to guess, something has changed in the version of jetty 
included with Solr 6.1.0, but I can't figure out what parameters are 
different.


Many thanks in advance for any assistance you can offer.

Re: Solr 6.1 Won't Start on Non-bash-Enabled System

[Yonik Seeley]

A unix system w/o bash is rare enough these days, it's unlikely to be supported.

As a workaround, last I checked, I was still able to create a WAR (by
jarring up the appropriate directory) and launch it the old 4.x way
(java -jar start.jar)
For your setup, try putting the webapp in /var/solr6/webapps/solr.war

-Yonik


On Wed, Aug 24, 2016 at 6:21 PM, Scott Vanderbilt  wrote:
> I'm attempting to upgrade a Solr 4.8.1 instance to 6.1. The host OS is
> OpenBSD 6.0.
>
> Attempting to start Solr (as in the on-line QuickStart) with:
>
>   bin/solr start -e cloud -noprompt
>
> fails with the error "env: bash: No such file or directory". Apparently
> bin/solr is a bash script. The OpenBSD host I'm using doesn't have bash
> installed (and unfortunately, no, I can't install it just for this). The
> default shell is ksh.
>
> When I try to use the script I previously used to start the 4.8.1 version, I
> get the warnings:
>
>WARNING: System properties and/or JVM args set.  Consider using
>  --dry-run or --exec
>WARNING: Nothing to start, exiting ..."
>
> The old startup script looks like this:
>
>JAVA_HOME=/usr/local/jdk-1.8.0 JAVA=java JETTY_HOME=/var/solr6
>  JETTY_PORT=8983 JETTY_LOGS=/var/solr6/logs
>  /usr/local/jdk-1.8.0/bin/java -Dsolr.solr.home=/var/solr6/solr
>  -Djava.util.logging.config.file=etc/logging.properties
>  -DSTOP.PORT=8079 -DSTOP.KEY=abc123 -jar /var/solr6/start.jar
>  --daemon &
>
> If I had to guess, something has changed in the version of jetty included
> with Solr 6.1.0, but I can't figure out what parameters are different.
>
> Many thanks in advance for any assistance you can offer.
>

Re: Solr 6.1 Won't Start on Non-bash-Enabled System

[Alexandre Rafalovitch]

Workarounds

1. get a VM with openness and bash.
2. Enable +x (echo expanded commands) on the first line of the script where
it specifies bash
3. Run the script with 'tee' or 'transcript' or other output redirect. Use
the full JVM launching command to see full expanded parameter list.

Or get Solr running and look at full process execution string.

Or ask somebody else who has OpenBSD with bash to do same.

Or try editing bash to ksh and see how you far you get, but I suspect not
very far, the conditional formats are different I believe.

Regards,
Alex

On 25 Aug 2016 5:22 AM, "Scott Vanderbilt"  wrote:

> I'm attempting to upgrade a Solr 4.8.1 instance to 6.1. The host OS is
> OpenBSD 6.0.
>
> Attempting to start Solr (as in the on-line QuickStart) with:
>
>   bin/solr start -e cloud -noprompt
>
> fails with the error "env: bash: No such file or directory". Apparently
> bin/solr is a bash script. The OpenBSD host I'm using doesn't have bash
> installed (and unfortunately, no, I can't install it just for this). The
> default shell is ksh.
>
> When I try to use the script I previously used to start the 4.8.1 version,
> I get the warnings:
>
>WARNING: System properties and/or JVM args set.  Consider using
>  --dry-run or --exec
>WARNING: Nothing to start, exiting ..."
>
> The old startup script looks like this:
>
>JAVA_HOME=/usr/local/jdk-1.8.0 JAVA=java JETTY_HOME=/var/solr6
>  JETTY_PORT=8983 JETTY_LOGS=/var/solr6/logs
>  /usr/local/jdk-1.8.0/bin/java -Dsolr.solr.home=/var/solr6/solr
>  -Djava.util.logging.config.file=etc/logging.properties
>  -DSTOP.PORT=8079 -DSTOP.KEY=abc123 -jar /var/solr6/start.jar
>  --daemon &
>
> If I had to guess, something has changed in the version of jetty included
> with Solr 6.1.0, but I can't figure out what parameters are different.
>
> Many thanks in advance for any assistance you can offer.
>
>

Re: Solr 6.1 Won't Start on Non-bash-Enabled System

[Scott Vanderbilt]

Thank you for your reply.

After some experimentation and reading of jetty docs, I found that by 
deleting the "--daemon" argument and add "--module=http" to my startup 
script, I was able to get Solr running, albeit, with several warnings 
that I would like to resolve.


When you say "look at full process execution string", how would I do that?

Best wishes.


On 8/24/2016 4:31 PM, Alexandre Rafalovitch wrote:

Workarounds

1. get a VM with openness and bash.
2. Enable +x (echo expanded commands) on the first line of the script where
it specifies bash
3. Run the script with 'tee' or 'transcript' or other output redirect. Use
the full JVM launching command to see full expanded parameter list.

Or get Solr running and look at full process execution string.

Or ask somebody else who has OpenBSD with bash to do same.

Or try editing bash to ksh and see how you far you get, but I suspect not
very far, the conditional formats are different I believe.

Regards,
Alex

On 25 Aug 2016 5:22 AM, "Scott Vanderbilt"  wrote:


I'm attempting to upgrade a Solr 4.8.1 instance to 6.1. The host OS is
OpenBSD 6.0.

Attempting to start Solr (as in the on-line QuickStart) with:

  bin/solr start -e cloud -noprompt

fails with the error "env: bash: No such file or directory". Apparently
bin/solr is a bash script. The OpenBSD host I'm using doesn't have bash
installed (and unfortunately, no, I can't install it just for this). The
default shell is ksh.

When I try to use the script I previously used to start the 4.8.1 version,
I get the warnings:

   WARNING: System properties and/or JVM args set.  Consider using
 --dry-run or --exec
   WARNING: Nothing to start, exiting ..."

The old startup script looks like this:

   JAVA_HOME=/usr/local/jdk-1.8.0 JAVA=java JETTY_HOME=/var/solr6
 JETTY_PORT=8983 JETTY_LOGS=/var/solr6/logs
 /usr/local/jdk-1.8.0/bin/java -Dsolr.solr.home=/var/solr6/solr
 -Djava.util.logging.config.file=etc/logging.properties
 -DSTOP.PORT=8079 -DSTOP.KEY=abc123 -jar /var/solr6/start.jar
 --daemon &

If I had to guess, something has changed in the version of jetty included
with Solr 6.1.0, but I can't figure out what parameters are different.

Many thanks in advance for any assistance you can offer.

ask question about solr 6

[Hamed Gh]

hello,dear
 i have a question about solr 6 or 1.6

question: solr 6 : search query with spaces between other languages words

query:
../solr/human_info/select?q=title:دانش آموزان
گاج&rows=0&facet.pivot=title&facet=true&facet.pivot.mincount=1

دانش آموزان گاج -> into english means -> GAJ institution students

query response:

019title:دانش آموزان
گاجtitle0true1title

آموزان75529
titleدانش75529
titleگاج75529

---
"دانش آموزان گاج" or into english means "GAJ institution students"
doesn`t work
  GAJ\ institution\ students   doesn`t work
 (GAJ institution students)doesn`t work

i just want completion -> `دانش آموزان گاج` into english means -> `GAJ
institution students`.
also count that became repeat.

Re: Is it safe to upgrade an existing field to docvalues?

[Toke Eskildsen]

Alessandro Benedetti  wrote:
> So basically using your tool you build a copy of the index ( similar to
> what optimize does) without affecting the main index right ?

Yes. Your step-by-step is spot on.

In the end we re-indexed everything, because there were other issues with the 
index we wanted to fix, so DVEnabler is very much a limited implementation. I 
am sure that the conversion code can be made a lot faster. 

I could envision a better tool that just used a source index and a destination 
schema and did the conversion with no setup-fuss. One could also add other 
usable conversions besides DV-switching: Multi-value-fields with always 1 
value/doc could be changed to true single-value, and vice-versa. String fields 
with numerics could be converted to true numerics. etc. Most of the data is 
already in the index, so it is "just" a question of re-packing it.

> This is actually a useful tool when re-indexing could be extremely long.

Thank you.


Important note: I stated that I created that tool, which I apologize for. 
Thomas Egense and I wrote it jointly.

- Toke Eskildsen