HyperLogLog for Solr

2013-05-27 Thread J Mohamed Zahoor
Hi Has anyone tried using HLL for using finding unique values of a field in solr? I am planning to use them to facet count on certain fields to reduce memory footprint. ./Zahoor

Re: multiple cache for same field

2013-05-27 Thread J Mohamed Zahoor
e for there to be 1.8 billion unique elements in that index. > > On May 20, 2013, at 1:20 PM, Erick Erickson wrote: > >> Not sure, never had to worry about what they are.. >> >> On Mon, May 20, 2013 at 12:28 PM, J Mohamed Zahoor wrote: >>> >>> Wha

Re: multiple cache for same field

2013-05-20 Thread J Mohamed Zahoor
gt; _3ffm.*. Each such group represents one segment. The number of > segments changes with merging etc. > > Best > Erick > > On Mon, May 20, 2013 at 6:43 AM, J Mohamed Zahoor wrote: >> Hi >> >> Why is that lucene field cache has multiple entries for

cache disable through solrJ

2013-05-20 Thread J Mohamed Zahoor
Hi How do i disable cache (Solr FieldValueCache) for certain queries... using HTTP it can be done using {!cache=false}... how can i do it from solrj? ./zahoor

Re: Avoid loading Lucene's field cache for certain fields

2013-05-19 Thread J Mohamed Zahoor
I am using Solr 4.2.1 ./zahoor On 20-May-2013, at 11:48 AM, J Mohamed Zahoor wrote: > Hi > > I am trying to avoid loading some fields in Lucene's FieldCache. > > Is there a way to avoid loading certain failed in Lucene's FieldCache. > One way is to declare them

Avoid loading Lucene's field cache for certain fields

2013-05-19 Thread J Mohamed Zahoor
Hi I am trying to avoid loading some fields in Lucene's FieldCache. Is there a way to avoid loading certain failed in Lucene's FieldCache. One way is to declare them multivalued.. Is there any other way? ./zahoor

Re: Java heap space exception in 4.2.1

2013-05-18 Thread J Mohamed Zahoor
aah… was doing a facet on a double field which was having 6 decimal places… No surprise that the lucene cache got full… .z/ahoor On 17-May-2013, at 11:56 PM, J Mohamed Zahoor wrote: > Memory increase a lot with queries which have facets… > > > ./Zahoor > > > On 17-

Best query method

2013-05-18 Thread J Mohamed Zahoor
Hi I am using solr 4.2.1. My index has products from different stores with different attributes. If i want to get the count of all products which belongs to store X which is coloured red and is in-stock… My question is : Which way of querying is better in-terms of "performance" and "cache u

Re: Java heap space exception in 4.2.1

2013-05-17 Thread J Mohamed Zahoor
Memory increase a lot with queries which have facets… ./Zahoor On 17-May-2013, at 10:00 PM, Shawn Heisey wrote: > On 5/17/2013 1:17 AM, J Mohamed Zahoor wrote: >> I moved to 4.2.1 from 4.1 recently.. everything was working fine until i >> added few more stats query.. >

Re: Java heap space exception in 4.2.1

2013-05-17 Thread J Mohamed Zahoor
Hprof introspection shows that huge Double Array are using up 75% of heap space... which belongs to Lucen's FieldCache.. ./zahoor On 17-May-2013, at 12:47 PM, J Mohamed Zahoor wrote: > Hi > > I moved to 4.2.1 from 4.1 recently.. everything was working fine until i > add

Re: Solr 4 memory usage increase

2013-05-17 Thread J Mohamed Zahoor
I get the same issue in 1.7.0_09-b05 also. ./zahoor On 17-May-2013, at 12:07 PM, Walter Underwood wrote: > It is past time to get off of Java 6. That is dead. End of life. No more > updates, not even for security bugs. > > What version of Java 6? Some earlier versions had bad bugs that Solr

Java heap space exception in 4.2.1

2013-05-17 Thread J Mohamed Zahoor
Hi I moved to 4.2.1 from 4.1 recently.. everything was working fine until i added few more stats query.. Now i am getting this error frequently that solr does not run even for 2 minutes continuously. All 5GB is getting used instantaneously in few queries... SEVERE: null:java.lang.RuntimeExcep

Re: stats cache

2013-05-08 Thread J Mohamed Zahoor
ed, May 8, 2013 at 10:13 AM, J Mohamed Zahoor wrote: > >> Thanks.. i am caching in HTTP now.. >> >> ./zahoor >> >> >> On 08-May-2013, at 3:58 AM, Yonik Seeley wrote: >> >>> On Tue, May 7, 2013 at 12:48 PM, J Mohamed Zahoor >> wrote:

Re: stats cache

2013-05-08 Thread J Mohamed Zahoor
Thanks.. i am caching in HTTP now.. ./zahoor On 08-May-2013, at 3:58 AM, Yonik Seeley wrote: > On Tue, May 7, 2013 at 12:48 PM, J Mohamed Zahoor wrote: >> Hi >> >> I am computing lots of stats as part of a query… >> looks like the solr caching is not helping he

facet.pivot limit

2013-05-07 Thread J Mohamed Zahoor
Hi is there a limit for facet pivot like we have in facet.limit? ./zahoor

stats cache

2013-05-07 Thread J Mohamed Zahoor
Hi I am computing lots of stats as part of a query… looks like the solr caching is not helping here… Does solr caches stats of a query? ./zahoor

Re: Max http connections in CloudSolrServer

2013-04-22 Thread J Mohamed Zahoor
On 18-Apr-2013, at 9:43 PM, Shawn Heisey wrote: > Are you using the Jetty included with Solr, or a Jetty installed separately? I am using the Jetty that comes with Solr. > The Jetty included with Solr has a maxThreads value of 1 in its config. > The default would be closer to 200, an

Re: stats.facet not working for timestamp field

2013-04-21 Thread J Mohamed Zahoor
eld? > > Best > Erick > > On Thu, Apr 18, 2013 at 8:37 AM, J Mohamed Zahoor wrote: >> Hi >> >> I am using SOlr 4.1 with 6 shards. >> >> i want to find out some "price" stats for all the days in my index. >> I ended up using stats compon

Re: Max http connections in CloudSolrServer

2013-04-18 Thread J Mohamed Zahoor
I dont yet know if this is the reason... I am looking if jetty has some limit on accepting connections.. ./zahoor On 18-Apr-2013, at 12:52 PM, J Mohamed Zahoor wrote: > > Thanks for this. > The reason i asked this was.. when i fire 30 queries simultaneously from 30 > threa

stats.facet not working for timestamp field

2013-04-18 Thread J Mohamed Zahoor
Hi I am using SOlr 4.1 with 6 shards. i want to find out some "price" stats for all the days in my index. I ended up using stats component like "stats=true&stats.field=price&stats.facet=timestamp". but it throws up error like Invalid Date String:' #1;#0;#0;#0;'[my(#0;' My Question is :

Re: Max http connections in CloudSolrServer

2013-04-18 Thread J Mohamed Zahoor
Thanks for this. The reason i asked this was.. when i fire 30 queries simultaneously from 30 threads using the same CloudSolrServer instance, some queries gets fired after a delay.. sometime the delay is 30-50 seconds... In solr logs i can see.. 20+ queries get fired almost immediately... but s

Max http connections in CloudSolrServer

2013-04-17 Thread J Mohamed Zahoor
Hi I am pumping parallel select queries using CloudSolrServer. It looks like it can handle only certain no of max connections... my Question is, How many concurrent queries does a CloudSolrServer can handle? An old thread tries to answer this by asking to give our own instance of LBHttpSolrSe

zkState changes too often

2013-04-16 Thread J Mohamed Zahoor
Hi I am using SolrCloud (4.1) with 6 nodes. When i index the documents from the mapper and as the load increases.. i see these messages in my mapper logs… WHich looks like it is slowing down my indexing speed. 2013-04-16 06:04:18,013 INFO org.apache.solr.common.cloud.ZkStateReader: Updating li

Re: CloudSolrServer vs ConcurrentUpdateSolrServer for indexing

2013-04-16 Thread J Mohamed Zahoor
It sure increased the performance . Thanks for the input. ./zahoor On 14-Apr-2013, at 10:13 PM, J Mohamed Zahoor wrote: > Thanks.. > Will try multithreading with CloudSolrServer. > > ./zahoor > > On 13-Apr-2013, at 9:11 PM, Mark Miller wrote: > >> >>

Re: CloudSolrServer vs ConcurrentUpdateSolrServer for indexing

2013-04-14 Thread J Mohamed Zahoor
Thanks.. Will try multithreading with CloudSolrServer. ./zahoor On 13-Apr-2013, at 9:11 PM, Mark Miller wrote: > > On Apr 13, 2013, at 11:07 AM, J Mohamed Zahoor wrote: > >> Hi >> >> This question has come up many times in the list with lots of variations

CloudSolrServer vs ConcurrentUpdateSolrServer for indexing

2013-04-13 Thread J Mohamed Zahoor
Hi This question has come up many times in the list with lots of variations (which confuses me a lot). Iam using Solr 4.1. one collection , 6 shards, 6 machines. I am using CloudSolrServer inside each mapper to index my documents…. While it is working fine , iam trying to improve the indexing

Query using function query result

2013-04-02 Thread J Mohamed Zahoor
Hi i want to query documents which match a certain dynamic criteria. like, How do i get all documents, where sub(field1,field2) < 0 ? i tried _val_: sub(field1,field2) and used fq:[_val_:[0 TO *] But it doesnt work. ./Zahoor

Re: multicore vs multi collection

2013-03-26 Thread J Mohamed Zahoor
> Solr setup to SolrCloud, > you would have 2 collections, each made up of multiple individual > SolrCores<http://wiki.apache.org/solr/SolrCores>. > > > 2013/3/26 J Mohamed Zahoor > >> Hi >> >> I am kind of confuzed between multi core and multi colle

multicore vs multi collection

2013-03-26 Thread J Mohamed Zahoor
Hi I am kind of confuzed between multi core and multi collection. Docs dont seem to clarify this.. can someone enlighten me what is ther difference between a core and a collection? Are they same? ./zahoor

Accessing multicore setup using solrj

2013-03-26 Thread J Mohamed Zahoor
Hi I am having a multi core setup with 2 core "core0" and core1". How do i insert doc in core 1? I am using as below. searchServer = new CloudSolrServer(zooQourumUrl); searchServer.setDefaultCollection("core1"); searchServer.connect(); and i get "No live solr servers" exception. But i could s

Re: NPE when adding docs in 4.2

2013-03-16 Thread J Mohamed Zahoor
Got it.. Thanks. ./Zahoor On 16-Mar-2013, at 9:13 PM, Yonik Seeley wrote: > On Sat, Mar 16, 2013 at 11:36 AM, J Mohamed Zahoor wrote: >> aahha… i used a replication factor of 0. >> I thought 0 means no replication of original.. >> >> Should that be 1 if i want no

Re: NPE when adding docs in 4.2

2013-03-16 Thread J Mohamed Zahoor
out not finding the collection rather than an NPE (for 4.3). > > - Mark > > On Mar 16, 2013, at 9:47 AM, J Mohamed Zahoor wrote: > >> Hi >> >> I am using Solr 4.2. I created 2 collections by using >> >> >> "http://localhost:8983/solr/admi

NPE when adding docs in 4.2

2013-03-16 Thread J Mohamed Zahoor
Hi I am using Solr 4.2. I created 2 collections by using "http://localhost:8983/solr/admin/collections?action=CREATE&name=collection1&numShards=1&replicationFactor=0&maxShardsPerNode=1"; "http://localhost:8983/solr/admin/collections?action=CREATE&name=collection2&numShards=1&replicationFactor=0

Re: Exception while using CloudServer

2013-02-13 Thread J Mohamed Zahoor
Apologies... I was using 4.1 in solr server and 4.0 in solrj client which caused this problem. ./zahoor On 13-Feb-2013, at 4:08 PM, J Mohamed Zahoor wrote: > Hi > > I think the > > "router":"compositeId" > > value inside the cluster stat

Re: Exception while using CloudServer

2013-02-13 Thread J Mohamed Zahoor
Hi I think the "router":"compositeId" value inside the cluster state is creating this problem. ./Zahoor On 13-Feb-2013, at 4:06 PM, J Mohamed Zahoor wrote: > > I am using Solr 4.0. > > ./zahoor > > > On 13-Feb-2013, at 3:56 PM, J Mohamed Zahoo

Re: Exception while using CloudServer

2013-02-13 Thread J Mohamed Zahoor
I am using Solr 4.0. ./zahoor On 13-Feb-2013, at 3:56 PM, J Mohamed Zahoor wrote: > Hi > > I was trying to connect to solr cloud using CloudServer, I get the following > exception. > I tried clearing the zookeeper state and then restarting the solr instances, > st

Exception while using CloudServer

2013-02-13 Thread J Mohamed Zahoor
Hi I was trying to connect to solr cloud using CloudServer, I get the following exception. I tried clearing the zookeeper state and then restarting the solr instances, still i get the same exception. am i missing something? org.apache.solr.common.cloud.ZkStateReader: Updating cluster state fro

Re: LoadBalancing while adding documents

2013-02-12 Thread J Mohamed Zahoor
Ooh.. I dint know that there is CloudSolrServer. Thanks for the pointer. Will explore that. ./zahoor On 13-Feb-2013, at 11:49 AM, J Mohamed Zahoor wrote: > > On 13-Feb-2013, at 8:11 AM, Erick Erickson wrote: > >> Hold on here. LBHttpSolrServer should not be used for indexing

Re: LoadBalancing while adding documents

2013-02-12 Thread J Mohamed Zahoor
On 13-Feb-2013, at 8:11 AM, Erick Erickson wrote: > Hold on here. LBHttpSolrServer should not be used for indexing in a > Master/Slave setup, but in SolrCloud you may use it. Indeed, > CloudSolrServer uses LBHttpSolrServer under the covers. In SolrCloud mode, ConcurrentUpdateSolrServer will alr

LoadBalancing while adding documents

2013-02-12 Thread J Mohamed Zahoor
Hi I have multi shard replicated index spread across two machines. Once a week, i delete the entire index and create it from scratch. Today i am using ConcurrentUpdateSolrServer in solrj to add documents to the index. I want to add documents through both the servers.. to utilise the resources.

distinct count of facet field values

2013-02-04 Thread J Mohamed Zahoor
Hi Is it possible to get the distinct count of a given facet field in Solr? A query like this q=*:*&facet=true&facet.field=cat display the counts of all the unique categories present like electronics: 100 applicances:200 etc.. But if the list is big.. i dont want to get the entire list and

Re: Solr 4.1 UI fail to display result

2013-02-02 Thread J Mohamed Zahoor
It works In chrome though... ./Zahoor@iPhone On 02-Feb-2013, at 4:34 PM, J Mohamed Zahoor wrote: >> >> I'm not sure why .. but this sounds like the JSON Parser was called with an >> HTML- or XML-String? After you hit the "Execute" Button on the Website, on

Re: Solr 4.1 UI fail to display result

2013-02-02 Thread J Mohamed Zahoor
> > I'm not sure why .. but this sounds like the JSON Parser was called with an > HTML- or XML-String? After you hit the "Execute" Button on the Website, on > the top of the right content-area, there is a link - which is what the UI > will request .. if you open that in another browser-tab or w

Image retrieval

2013-02-01 Thread J Mohamed Zahoor
Hi Iam trying to integrate a image retrieval system (Lire - http://www.semanticmetadata.net/lire/) in to Solr. Images are stored as its descriptors or features. But when searching for a image match, I need to pick images whose features are close (by calculating the distance) to the given image.

Re: Solr 4.1 UI fail to display result

2013-01-30 Thread J Mohamed Zahoor
LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch > - Time is the quality of nature that keeps events from happening all at > once. Lately, it doesn't seem to be working. (Anonymous - via GTD book) > > > On Wed, Jan 30, 2013 at 8:17 AM, J Mohamed Zahoor wrote: > >> The stac

Re: Solr 4.1 UI fail to display result

2013-01-30 Thread J Mohamed Zahoor
The stack is format_json -- app.js (465) json -- query.js (59) complete - query.js (77) fire -- require.js (3099) fireWith -- require.js (3217) done -- require.js (9469) callback -- require.js (10235) ./zahoor On 30-Jan-2013, at 6:43 PM, J Mohamed Zahoor wrote: > Hi > > Iam

Solr 4.1 UI fail to display result

2013-01-30 Thread J Mohamed Zahoor
Hi Iam using 4.1 release and i see a problem when i set the response type as JSON in the UI. I am using Safari 6.0.2 and i see a "SyntaxError: JSON Parse error: Unrecognized token '<'". app.js line 465. When i debug more.. i see the response is still coming in XML format. Is anyone else fa

Re: zookeeper config

2013-01-24 Thread J Mohamed Zahoor
Cool. Thanks. On 24-Jan-2013, at 1:28 PM, Per Steffensen wrote: > This is supported. You just need to ajust your ZK connection-string: > ":/solr,:/solr,...,:/solr" > > Regards, Per Steffensen > > On 1/24/13 7:57 AM, J Mohamed Zahoor wrote: >> Hi >> &

zookeeper config

2013-01-23 Thread J Mohamed Zahoor
Hi I am using Solr 4.0. I see the Solr data in zookeeper is placed on the root znode itself. This becomes a pain if the zookeeper instance is used for multiple projects like HBase and like. I am thinking of raising a Jira for putting them under a znode /solr or something like that? ./Zahoor

Re: retrieving latest document **only**

2013-01-16 Thread J Mohamed Zahoor
group field is timestamp… it is not multivalued. ./zahoor On 15-Jan-2013, at 7:14 PM, Upayavira wrote: > Is your group field multivalued? Could docs appear in more than one > group? > > Upayavira > > On Tue, Jan 15, 2013, at 01:22 PM, J Mohamed Zahoor wrote: >> >

Re: retrieving latest document **only**

2013-01-15 Thread J Mohamed Zahoor
The sum of all the "count" in the groups… does not match the total no of docs found. ./zahoor On 12-Jan-2013, at 1:27 PM, Upayavira wrote: > Not sure exactly what you mean, can you give an example? > > Upayavira > > On Sat, Jan 12, 2013, at 06:32 AM, J Mohamed

Re: retrieving latest document **only**

2013-01-11 Thread J Mohamed Zahoor
Cool… it worked… But the count of all the groups and the count inside stats component does not match… Is that a bug? ./zahoor On 11-Jan-2013, at 6:48 PM, Upayavira wrote: > could you use field collapsing? Boost by date and only show one value > per group, and you'll have the most recent docum

4.1 release date

2012-12-28 Thread J Mohamed Zahoor
Hi When can we expect 4.1? ./zahoor

Re: multi field query with selective results

2012-12-23 Thread J Mohamed Zahoor
I don't think I hijacked any thread. it is a new thread. Can you please enlighten me? On Sunday, December 23, 2012, Lance Norskog wrote: > Please start a new thread. > > Thanks! > > On 12/22/2012 11:03 AM, J Mohamed Zahoor wrote: > >> Hi >> >> I have a

multi field query with selective results

2012-12-22 Thread J Mohamed Zahoor
Hi I have a word completion requirement where i need to pick result from two indexed fields. The trick is i need to pick top 5 results from each field and display as suggestions. If i set fq as field1:XXX AND field2:XXX, the top result comes entirely from field1 matches. Is there any other way

Faceting on Dynamic fields

2012-12-17 Thread Mohamed Zahoor
Hi I have many dynamic fields in my schema name_X where X can range from 0 to 10. Not all documents will have all the fields from 0 to 10. I want to do a fecet on these fields. I have seen SOLR-247 and other queries in this list. Is there any other way other than patching SOLR-247 on 4.0? ./Z