Re: Are there roadblocks to creating custom DocRouter implementations?

Also interested in custom/pluggable routing. On Tue, May 16, 2017 at 4:47 PM, Erick Erickson wrote: > Hmmm, would the functionality be served by just using implicit routing > putting the logic in creating the doc and populating the route field? > Not, perhaps, as elegant as having some kind of p

Re: knowing which fields were successfully hit

Something like elasticsearch named-queries, right https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-named-queries-and-filters.html ? On Tue, May 16, 2017 at 7:10 PM, John Blythe wrote: > sorry for the confusion. as in i received results due to matches on field x > v

Re: TrieIntField vs IntPointField performance only for equality comparison (no range filtering)

Hi Shawn, I forgot that legacy-int-fields were deprecated. Point fields it is then. Thanks, Dorian On Tue, May 16, 2017 at 3:01 PM, Shawn Heisey wrote: > On 5/16/2017 3:33 AM, Dorian Hoxha wrote: > > Has anyone measured which is more efficient/performant between the 2 > >

TrieIntField vs IntPointField performance only for equality comparison (no range filtering)

Hi, Has anyone measured which is more efficient/performant between the 2 intfields if we don't need to do range-checking ? (precisionStep=0) Regards, Dorian

Re: Atomic Updates

> > > wrote: > > > > > Hmm, interesting. I can imagine that as long as you're updating > > > docValues fields, the other_text field would be there. But the instant > > > you updated a non-docValues field (text_field in your example) the > > > other_

Re: Atomic Updates

ioned in the first > post, I've already tried a few tests, and the value seems to still be > present after an atomic update. > > I haven't exhausted all possible atomic updates, but 'set' and 'add' seem > to preserve the non-stored text field. > > Than

Re: Atomic Updates

You'll lose the data in that field. Try doing a commit and it should happen. On Wed, Apr 26, 2017 at 9:50 PM, Chris Ulicny wrote: > Thanks Shawn, I didn't realize docValues were enabled by default now. > That's very convenient and probably makes a lot of the schemas we've been > making excessive

Re: Return all docs with same last-value when sorting by non-unique-value

Alex > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 15 April 2017 at 08:06, Dorian Hoxha wrote: > > Hi friends, > > > > Say we're sorting by a non-unique-value, and also have a limit(x). But > > there are m

Return all docs with same last-value when sorting by non-unique-value

Hi friends, Say we're sorting by a non-unique-value, and also have a limit(x). But there are more docs in the end of list(x) that have the same value. Is it possible to return them even if the number of items will be > x ? This will make it possible so I don't have to sort by (non-unique,unique) v

Re: Filtering results by minimum relevancy score

@alessandro Elastic-search has it: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-min-score.html On Wed, Apr 12, 2017 at 1:49 PM, alessandro.benedetti wrote: > I am not completely sure that the potential benefit of merging less docs in > sharded pagination overcom

Re: Dynamic schema memory consumption

apache.org/jira/browse/SOLR-7191 On Tue, Apr 11, 2017 at 9:11 PM, Dorian Hoxha wrote: > And this overhead depends on what? I mean, if I create an empty collection >> will it take up much heap size just for "being there" ? > > Yes. You can search on elastic-search

Re: Dynamic schema memory consumption

s will have a schema and some data/segments and translog. On Tue, Apr 11, 2017 at 7:41 PM, jpereira wrote: > The way the data is spread across the cluster is not really uniform. Most > of > shards have way lower than 50GB; I would say about 15% of the total shards > have more than 50G

Re: Dynamic schema memory consumption

ow about static/dynamic memory-issue though. On Tue, Apr 11, 2017 at 6:09 PM, jpereira wrote: > Dorian Hoxha wrote > > Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a > > little too much for 3TB of data ? > > Something like 0.167GB for each shard

Re: Filtering results by minimum relevancy score

Can't the filter be used in cases when you're paginating in sharded-scenario ? So if you do limit=10, offset=10, each shard will return 20 docs ? While if you do limit=10, _score<=last_page.min_score, then each shard will return 10 docs ? (they will still score all docs, but merging will be faster)

Re: Dynamic schema memory consumption

Also you should change the heap 32GB->30GB so you're guaranteed to get pointer compression. I think you should have no need to increase it more than this, since most things have moved to out-of-heap stuff, like docValues etc. On Tue, Apr 11, 2017 at 12:07 PM, Dorian Hoxha wrote: >

Re: Dynamic schema memory consumption

Isn't 18K lucene-indexes (1 for each shard, not counting the replicas) a little too much for 3TB of data ? Something like 0.167GB for each shard ? Isn't that too much overhead (i've mostly worked with es but still lucene underneath) ? Can't you use 1/100 the current number of collections ? On Mon

Re: SortingMergePolicy in solr 6.4.2

Did you get any update on this ? On Tue, Mar 14, 2017 at 11:56 AM, Sahil Agarwal wrote: > The SortingMergePolicy does not seem to get implemeted. > > The csv file gets indexed without errors. But when I search for a term, the > results returned are not sorted by Marks. > > Following is a toy pro

Str vs Int/Long uniqueKey field performance

Hey friends, Is there any difference on the index-size/performance of having the 'uniqueKey` field a long vs a string ? Meaning, does it use a different data-structure ? Cause I remember elasticsearch always uses a string (since it adds the #type, which solr doesn't have). Regards, Dorian

Re: Performance degradation after upgrading from 6.2.1 to 6.4.1

Did you see the other thread ? It looked like a problem with logging. On Tue, Feb 14, 2017 at 10:52 AM, Henrik Brautaset Aronsen < henrik.aron...@gmail.com> wrote: > We are seeing performance degradation on our SolrCloud instances after > upgrading to 6.4.1. > > > Here are a couple of graphs. As

Re: solr-user-unsubscribe

Come on dude. Just look at instructions. Have a little respect. On Mon, Jan 30, 2017 at 1:55 PM, Rowe, William - 1180 - MITLL < william.r...@ll.mit.edu> wrote: > solr-user-unsubscribe > > >

Commit required after delete ?

Hello friends, Based on what I've read, I think "commit" isn't needed to make deletes active (like we do with index/update), right ? Since it just marks an in-memory deleted-id bitmap, right ? Thank You

Re: Cloud Behavior when using numShards=1

I think solr tries itself to load balance. Read this page https://cwiki.apache.org/confluence/display/solr/Distributed+Requests (preferLocalShards!) Also please write the query. tip: fill "send" address after completing email On Tue, Dec 27, 2016 at 4:31 PM, Dave Seltzer wrote: > [Forgive the

Re: ttl on merge-time possible somehow ?

On Mon, Dec 19, 2016 at 7:03 PM, Chris Hostetter wrote: > > : So, the other way this can be made better in my opinion is (if the > : optimization is not already there) > : Is to make the 'delete-query' on ttl-documents operation on translog to > not > : be forced to fsync to disk (so still wr

Re: Has anyone used linode.com to run Solr | ??Best way to deliver PHP/Apache clients with Solr question

On Sun, Dec 18, 2016 at 3:48 PM, GW wrote: > Yeah, > > > I'll look at the proxy you suggested shortly. > > I've discovered that the idea of making a zookeeper aware app is pointless > when scripting REST calls right after I installed libzookeeper. > > Zookeeper is there to provide the zookeeping

Re: Soft commit and reading data just after the commit

There's a very high probability that you're using the wrong tool for the job if you need 1ms softCommit time. Especially when you always need it (ex there are apps where you need commit-after-insert very rarely). So explain what you're using it for ? On Sun, Dec 18, 2016 at 3:38 PM, Lasitha Watta

Re: ttl on merge-time possible somehow ?

On Sat, Dec 17, 2016 at 12:04 AM, Chris Hostetter wrote: > > : > lucene, something has to "mark" the segements as deleted in order for > them > ... > : Note, it doesn't mark the "segment", it marks the "document". > > correct, typo on my part -- sorry. > > : > The disatisfaction you expre

Re: ttl on merge-time possible somehow ?

On Fri, Dec 16, 2016 at 10:53 PM, Chris Hostetter wrote: > > : Yep, that's what came in my search. See how TTL work in hbase/cassandra/ > : rocksdb . There > : isn't a "delete old docs"query, but old docs are deleted by the storage > : when m

Re: ttl on merge-time possible somehow ?

anks Shawn. On Fri, Dec 16, 2016 at 9:57 PM, Shawn Heisey wrote: > On 12/16/2016 1:12 PM, Dorian Hoxha wrote: > > Shawn, I know how it works, I read the blog post. But I don't want it > > that > > way. So how to do it my way? Like a custom merge function on lucene or >

Re: ttl on merge-time possible somehow ?

On Fri, Dec 16, 2016 at 8:11 PM, Shawn Heisey wrote: > On 12/16/2016 11:13 AM, Dorian Hoxha wrote: > > Yep, that's what came in my search. See how TTL work in hbase/cassandra/ > > rocksdb <https://github.com/facebook/rocksdb/wiki/Time-to-Live>. There > > isn

Re: Max vertical scaling in your experience ? (1 instance/server)

d is on a real cluster not on a SAN. > > > > The bit about dead with no error. I got this from a post I made asking > > about the best way to deploy apps. Was shown some code on making your app > > zookeeper aware. I am just getting to this so I'm talking from my ass. A &g

Re: ttl on merge-time possible somehow ?

On Fri, Dec 16, 2016 at 4:42 PM, Shawn Heisey wrote: > On 12/16/2016 12:54 AM, Dorian Hoxha wrote: > > I did some search for TTL on solr, and found only a way to do it with > > a delete-query. But that ~sucks, because you have to do a lot of > > inserts (and queries). > &

Re: Separating Search and Indexing in SolrCloud

is not a question of SolrClient cause that is clear how to direct > search request to specific nodes. This is more about index optimization > so that certain nodes (ie. replicas) could suffer less due to high > volume indexing while serving search requests. > > > > > On 16/12/1

Re: Separating Search and Indexing in SolrCloud

The leader is the source of truth. You expect to make the replica the source of truth or something???Doesn't make sense? What people do, is send write to leader/master and reads to replicas/slaves in other solr/other-dbs. On Fri, Dec 16, 2016 at 1:31 PM, Jaroslaw Rozanski wrote: > Hi all, > > Ac

Re: Max vertical scaling in your experience ? (1 instance/server)

;other commenter wrote on your thread. > > Will keep you posted on my roll out if you like. > > Best, > > GW > > > > > > > > On 16 December 2016 at 03:31, Dorian Hoxha wrote: > > > Hello searchers, > > > > I'm researching solr for a pr

Re: Max vertical scaling in your experience ? (1 instance/server)

On Fri, Dec 16, 2016 at 11:31 AM, Toke Eskildsen wrote: > On Fri, 2016-12-16 at 11:19 +0100, Dorian Hoxha wrote: > > On Fri, Dec 16, 2016 at 10:45 AM, Toke Eskildsen > > wrote: > > > We try hard to stay below 32GB, but for some setups the penalty of > > > cros

Re: Max vertical scaling in your experience ? (1 instance/server)

On Fri, Dec 16, 2016 at 10:45 AM, Toke Eskildsen wrote: > On Fri, 2016-12-16 at 09:31 +0100, Dorian Hoxha wrote: > > I'm researching solr for a project that would require a max- > > inserts(10M/s) and some heavy facet+fq on top of that, though on low > > qps. > >

Max vertical scaling in your experience ? (1 instance/server)

Hello searchers, I'm researching solr for a project that would require a max-inserts(10M/s) and some heavy facet+fq on top of that, though on low qps. And I'm trying to find blogs/slides where people have used some big machines instead of hundreds of small ones. 1. Largest I've found is this

ttl on merge-time possible somehow ?

Hello searchers, I did some search for TTL on solr, and found only a way to do it with a delete-query. But that ~sucks, because you have to do a lot of inserts (and queries). The other(kinda better) way to do it, is to set a collection-level ttl, and when indexes are merged, they will drop the do

Re: Search only for single value of Solr multivalue field

You should be able to filter "(word1 in field OR word2 in field) AND NOT(word1 in field AND word2 in field)". Translate that into the right syntax. I don't know if lucene is smart enough to execute the filter only once (it should be i guess). Makes sense ? On Thu, Dec 15, 2016 at 12:12 PM, Leo BRU

Re: Has anyone used linode.com to run Solr | ??Best way to deliver PHP/Apache clients with Solr question

stances. > I've done that before (though with es, but it's ~same). And just contacting the localhost solr. The problem with that, is that if the solr on the current host fails, your php won't work. So best in this scenario is to have an array of hosts, but the first being the local sol

Re: Has anyone used linode.com to run Solr | ??Best way to deliver PHP/Apache clients with Solr question

See replies inline: On Wed, Dec 14, 2016 at 11:16 AM, GW wrote: > Hello folks, > > I'm about to set up a Web service I created with PHP/Apache <--> Solr Cloud > > I'm hoping to index a bazillion documents. > ok , how many inserts/second ? > > I'm thinking about using Linode.com because the pric

Re: Difference between currency fieldType and float fieldType

sther Quansah > > > Le 7 déc. 2016 à 07:26, Dorian Hoxha a écrit : > > > > Come on dude, just use the int/long. > > Source: double is still a float. > > > > On Wed, Dec 7, 2016 at 1:17 PM, Zheng Lin Edwin Yeo < > edwinye...@gmail.com> > > wrote: >

Re: Difference between currency fieldType and float fieldType

will it hit the same issue again if the amount exceeds 64-bit? > > Regards, > Edwin > > > On 7 December 2016 at 15:28, Dorian Hoxha wrote: > > > Yeah, you'll have to do the conversion yourself (or something internal, > > like the currencyField). > > > >

Re: Difference between currency fieldType and float fieldType

Yeah, you'll have to do the conversion yourself (or something internal, like the currencyField). Think about it as datetimes. You store everything in utc (cents), but display to each user in it's own timezone (different currency, or just from cents to full dollars). On Wed, Dec 7, 2016 at 8:23 AM

Re: Difference between currency fieldType and float fieldType

Don't use float for money (in whatever db). https://wiki.apache.org/solr/CurrencyField What you do is save the money as cents, and store that in a long. That's what the currencyField probably does for you inside. It provides currency conversion at query-time. On Tue, Dec 6, 2016 at 4:45 AM, Zheng

Re: Queries regarding solr cache

@Shawn Any idea why the cache doesn't use roaring bitsets ? On Thu, Dec 1, 2016 at 3:49 PM, Shawn Heisey wrote: > On 12/1/2016 4:04 AM, kshitij tyagi wrote: > > I am using Solr and serving huge number of requests in my application. > > > > I need to know how can I utilize caching in Solr. > > >

Realtime multi get with different (_route_, fields, etc) for each id

Hello searchers, Looks like this is not possible, right ? It means I have to specify all the _route_ in the request, and each shard will try to lookup all the ids internally. Is there a way to specify it ? Like elasticsearch does https://www.elastic.co/guide/en/elasticsearch/reference/current/do

Re: Index time sorting and per index mergePolicyFactory

bump after 11 days On Thu, Nov 17, 2016 at 10:25 AM, Dorian Hoxha wrote: > Hi, > > I know this is done in lucene, but I don't see it in solr (by searching + > docs on collections). > > I see https://cwiki.apache.org/confluence/display/solr/ > IndexConfig+in+SolrConfig

Re: update a document without changing anything

Thanks Ishan. On Sun, Nov 27, 2016 at 6:42 AM, Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > Maybe do an "inc" of 0 to a numeric field for every document. > https://cwiki.apache.org/confluence/display/solr/ > Updating+Parts+of+Documents > > On Wed,

Re: Best python 3 client for solrcloud

gt; > Check it out and let me know what you think. > > On Nov 24, 2016 3:51 PM, "Dorian Hoxha" wrote: > > > Hi searchers, > > > > I see multiple clients for solr in python but each one looks like misses > > many features. What I need is for at least th

Best python 3 client for solrcloud

Hi searchers, I see multiple clients for solr in python but each one looks like misses many features. What I need is for at least the low-level api to work with cloud (like retries on different nodes and nice exceptions). What is the best that you use currently ? Thank You!

Re: Should zookeeper be run on the worker machines?

You can, but you should not. Source: heavy load may slow zookeeper resulting in timeouts etc. On Wed, Nov 23, 2016 at 5:00 PM, Tech Id wrote: > Hi, > > Can someone please respond to this zookeeper-for-Solr Stack-Overflow > question: http://stackoverflow.com/questions/40755137/should- > zookeeper

update a document without changing anything

Hello searcherers, So, I have document that is fully stored. Then I make small change in schema. And now I have to reinsert every document. But I'm afraid of doing a get+insert, because something else may change the document in the meantime. So I want to do an "update" of nothing, so internally on

Re: Solr/lucene "planet" + recommendations for blogs to follow

#x27;ll figure it out. I > have enough info. > > Regards, >Alex. > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 22 November 2016 at 22:45, Dorian Hoxha wrote: > > Thanks Alex, some kind of weekly newsletter w

Re: Solr/lucene "planet" + recommendations for blogs to follow

https://lucidworks.com/blog/ > > Regards, >Alex. > > > http://www.solr-start.com/ - Resources for Solr users, new and experienced > > > On 22 November 2016 at 21:56, Dorian Hoxha wrote: > > Hello searcherers, > > > > Is there a solr/lucene "planet" like planet.postgresql.org ? If not, > what > > are some blogs/rss/feeds that I should follow to learn what's happening > in > > the solr/lucene worlds ? > > > > Thank You >

Re: Multiple search-queries in 1 http request ?

; > >> Personally, I think starting multiple queries asynchronously right after > >> each other has little disadvantages over a batching mechanism. > >> > >> Best regards, > >> Chris > >> > >> > >> [0] https://issues.apache.org/ji

Solr/lucene "planet" + recommendations for blogs to follow

Hello searcherers, Is there a solr/lucene "planet" like planet.postgresql.org ? If not, what are some blogs/rss/feeds that I should follow to learn what's happening in the solr/lucene worlds ? Thank You

Re: Using solr(cloud) as source-of-truth for data (with no backing external db)

to save a DB lookup. > > I would still store everything in a DB though to have a "master" copy of > everthing. > > > > On 11/18/2016 04:45 AM, Dorian Hoxha wrote: > >> @alex >> That makes sense, but it can be ~fixed by just storing every field that >&

Re: Bkd tree numbers/geo on solr 6.3 ?

Looks like it needs https://issues.apache.org/jira/browse/SOLR-8396 . On Thu, Nov 17, 2016 at 2:41 PM, Dorian Hoxha wrote: > Hi, > > I've read that lucene 6 has fancy bkd-tree implementation for numbers. But > on latest cwiki I only see TrieNumbers. Aren't they imple

Re: Using solr(cloud) as source-of-truth for data (with no backing external db)

, > > Alex. > > > > Solr Example reading group is starting November 2016, join us at > > http://j.mp/SolrERG > > Newsletter and resources for Solr beginners and intermediates: > > http://www.solr-start.com/ > > > > > > On 17 November 2016 at 18:46, Dorian Hoxha > wrote: > >> Hi, > >> > >> Anyone use solr for source-of-data with no `normal` db (of course with > >> normal backups/replication) ? > >> > >> Are there any drawbacks ? > >> > >> Thank You > >

Changing route and still ending on the same shard (or increasing the % of shards a tenant is distrbuted on without reindexing)

Hi, Assuming I use `tenant1/4!doc50` for id (which means 1/16th of shards), and I later change it to `tenant1/2!doc50` (which means 1/8), is it guaranteed that the document will go to the same shard ? (it would be nice, but I don't think so). Meaning , when you change the `/x!`, do you have to rei

Bkd tree numbers/geo on solr 6.3 ?

Hi, I've read that lucene 6 has fancy bkd-tree implementation for numbers. But on latest cwiki I only see TrieNumbers. Aren't they implemented or did I miss something (they still mention "indexing multiple values for range-queries" , which is the old way)? Thank You

Multiple search-queries in 1 http request ?

Hi, I couldn't find anything in core for "multiple separate queries in 1 http request" like elasticsearch ? I found this blog-post t

Updating documents with docvalues (not stored), commit question

Looks like you can update documents even using just doc-values (without stored). While I understand the columnar-format, my issue with this is that docValues are added when a 'commit' is done (right?). Does that mean that it will force a commit (which is a slow operation) when updating with docValu

Re: Parent child relationship, where children aren't nested but separate (like elasticsearch)

Solr Example reading group is starting November 2016, join us at > http://j.mp/SolrERG > Newsletter and resources for Solr beginners and intermediates: > http://www.solr-start.com/ > > > On 17 November 2016 at 18:08, Dorian Hoxha wrote: > > Hi, > > > > I'm no

Re: "add and limit" update modifier or scripted update like elasticsearch

ovember 2016, join us at > http://j.mp/SolrERG > Newsletter and resources for Solr beginners and intermediates: > http://www.solr-start.com/ > > > On 17 November 2016 at 18:06, Dorian Hoxha wrote: > > Hi, > > > > Is there an "add and limit" update modifier

Index time sorting and per index mergePolicyFactory

Hi, I know this is done in lucene, but I don't see it in solr (by searching + docs on collections). I see https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig but it's not mentioned for index-time-sorting. So, is it possible and definable for each index ? I want to have som

Using solr(cloud) as source-of-truth for data (with no backing external db)

Hi, Anyone use solr for source-of-data with no `normal` db (of course with normal backups/replication) ? Are there any drawbacks ? Thank You

How many versions do you stay behind in production for better stability ?

Hi, I see that there is a new release on every lucene release. Do you always use the latest version since it may have bugs (ex most cassandra productions are old compared to latest `stable` version because they're not stable). How much behind do you usually stay ? (ex: 6.3 just came out, and you n

Parent child relationship, where children aren't nested but separate (like elasticsearch)

Hi, I'm not finding a way to support parent-child like es does (using blockjoin)? I've seen some blogs with having children as nested inside the parent-document, but I want to freely crud childs/parents as sepa

"add and limit" update modifier or scripted update like elasticsearch

Hi, Is there an "add and limit" update modifier (couldn't find in docs) ? If not, can I run a script to update a document (still couldn't find anything) ? If not, how should I do that (custom plugin? )? Thank You

Re: book for Solr 3.4?

@HelponR Curious why you're interested in an old version ? On Tue, Nov 15, 2016 at 11:43 PM, HelponR wrote: > Thank you. Just found one here https://wiki.apache.org/solr/SolrResources > > "Apache Solr 3 Enterprise Search Server > by David Smiley and