Broken stats.js

[Mark Mandel]

Relatively new to SOLR (only JUST deployed my first SOLR app to production,
very proud ;o) )

I went to check out the solr/mycore/admin/stats.jsp page... and all I get is
a blank page.

Looking into it deeper, it seems that SOLR is returning badly encoded XML to
the browser, so it's not rendering.

I can't seem to find any references to this issue anywhere except :
https://issues.apache.org/jira/browse/SOLR-1750

(Which has more of a workaround), and it seems that the SolrInfoMBeanHandler
is not in the 1.4.1 build.

Any help would be appreciated, so I can tune the caching settings on my SOLR
install (which so far is screaming along, but it's always good to have more
speed).

Thanks in advance,

Mark

-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Broken stats.js

[Mark Mandel]

Ah cool, thanks for your help.

I'll get digging, and see what I can do.

Mark

On Tue, Mar 29, 2011 at 11:36 AM, Chris Hostetter
wrote:

>
> : I can't seem to find any references to this issue anywhere except :
> : https://issues.apache.org/jira/browse/SOLR-1750
> :
> : (Which has more of a workaround), and it seems that the
> SolrInfoMBeanHandler
> : is not in the 1.4.1 build.
>
> correct, it will be in 3.1 however.
>
> it's not so much a workarround as it is a total abandonment of stats.jsp
> in favor of something that is easier to test, maintain, and use.
>
> : Any help would be appreciated, so I can tune the caching settings on my
> SOLR
> : install (which so far is screaming along, but it's always good to have
> more
> : speed).
>
> the one thing i can suggest that should work out of the box with solr
> 1.4.1 is to config solr to use JMX and then run a JMX client to query solr
> for those stats...
>
>http://wiki.apache.org/solr/SolrJmx
>
> ...that bypasses the stupid jsp completley.
>
> -Hoss
>



-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Search Regression Testing

[Mark Mandel]

Hey guys,

I'm wondering how people are managing regression testing, in particular with
things like text based search.

I.e. if you change how fields are indexed or change boosts in dismax,
ensuring that doesn't mean that critical queries are showing bad data.

The obvious answer to me was using unit tests. These may be brittle as some
index data can change over time, but I couldn't think of a better way.

How is everyone else solving this problem?

Cheers,

Mark

-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Search Regression Testing

[Mark Mandel]

Thanks for the input guys.

I've decided to implement some unit tests for now, although we don't have a
clean data set to work from (sucks, I know).

We're going to keep track of a set of vital queries, and ensure they don't
return 0 results, as we have a pretty decent level of confidence with Solr's
text matching. So not ideal, but better than nothing ;o)

That should find anything that's gone horribly wrong, while at the same time
dealing with our data set changing, and us not having very brittle tests.

Much appreciated,

Mark

On Wed, Apr 6, 2011 at 6:54 PM, Paul Libbrecht  wrote:

> Mark,
>
> In one project, with Lucene not Solr, I also use a smallish unit test
> sample and apply some queries there.
> It is very limited but is automatable.
>
> I find a better way is to have precision and recall measures of real users
> run release after release.
> I could never fully apply this yet on a recurring basis sadly.
>
> My ideal world would be that the search sample is small enough and that
> users are able to restrict search to this.
> Then users have the possibility of checking correctness of each result
> (say, first 10) for each query out of which one can then read results.
> Often, users provide comments along, e.g. missing matches. This is packed as
> a wiki page.
> First samples generally do not use enough of the features, this is adjusted
> as a dialogue.
>
> As a developer I review the test suite run and plan for next adjustments.
> The numeric approach allows easy mean precision and mean recall which is
> good for reporting.
>
> My best reference for PR testing and other forms of testing Kavi Mahesh's
> Text Retrieval Quality, a primer:
> http://www.oracle.com/technetwork/database/enterprise-edition/imt-quality-092464.html
>
> I would love to hear more of what the users have been doing.
>
> paul
>
>
> Le 6 avr. 2011 à 08:10, Mark Mandel a écrit :
>
> > Hey guys,
> >
> > I'm wondering how people are managing regression testing, in particular
> with
> > things like text based search.
> >
> > I.e. if you change how fields are indexed or change boosts in dismax,
> > ensuring that doesn't mean that critical queries are showing bad data.
> >
> > The obvious answer to me was using unit tests. These may be brittle as
> some
> > index data can change over time, but I couldn't think of a better way.
> >
> > How is everyone else solving this problem?
> >
> > Cheers,
> >
> > Mark
> >
> > --
> > E: mark.man...@gmail.com
> > T: http://www.twitter.com/neurotic
> > W: www.compoundtheory.com
> >
> > cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
> > http://www.cfobjective.com.au
> >
> > Hands-on ColdFusion ORM Training
> > www.ColdFusionOrmTraining.com
>
>


-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Allowing looser matches

[Mark Mandel]

Not sure if the title explains it all, or if what I want is even possible,
but figured I would ask.

Say, I have a series of products I'm selling, and a search of:

"Blue Wool Rugs"

Comes in.  This returns 0 results, as "Blue" and "Rugs" match terms that are
indexes, "Wool" does not.

Is there a way to configure my index/searchHandler, to either:

(a) if no documents are returned, look to partial matches of the search
(e.g. return results with 'Blue rugs', in this case)
(b) add results to the overall search, but at a lower score, that have only
*some* of the terms being searched in them (in this case, maybe 2/3)

Is that even possible?

Thanks,

Mark

-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Allowing looser matches

[Mark Mandel]

Thanks!

I searched high and low for that, couldn't see it in front of my face!

Mark

On Wed, Apr 13, 2011 at 6:32 PM, Pierre GOSSE wrote:

> For (a) I don't think anything exists today providing this mechanism.
> But (b) is a good description of the dismax handler with a MM parameter of
> 66%.
>
> Pierre
>
> -Message d'origine-
> De : Mark Mandel [mailto:mark.man...@gmail.com]
> Envoyé : mercredi 13 avril 2011 10:04
> À : solr-user@lucene.apache.org
> Objet : Allowing looser matches
>
> Not sure if the title explains it all, or if what I want is even possible,
> but figured I would ask.
>
> Say, I have a series of products I'm selling, and a search of:
>
> "Blue Wool Rugs"
>
> Comes in.  This returns 0 results, as "Blue" and "Rugs" match terms that
> are
> indexes, "Wool" does not.
>
> Is there a way to configure my index/searchHandler, to either:
>
> (a) if no documents are returned, look to partial matches of the search
> (e.g. return results with 'Blue rugs', in this case)
> (b) add results to the overall search, but at a lower score, that have only
> *some* of the terms being searched in them (in this case, maybe 2/3)
>
> Is that even possible?
>
> Thanks,
>
> Mark
>
> --
> E: mark.man...@gmail.com
> T: http://www.twitter.com/neurotic
> W: www.compoundtheory.com
>
> cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
> http://www.cfobjective.com.au
>
> Hands-on ColdFusion ORM Training
> www.ColdFusionOrmTraining.com
>



-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Facet sorting

[Mark Mandel]

Quick question -

Is it possible to sort some facets by Lex, and others by numeric value?

>From everything I've seen, it's either all or none, with no in between. Was
just curious if that was not the case.

Getting around it by doing post-sorting with comparators, but thought I may
not need that step.

Cheers,

Mark

-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Facet sorting

[Mark Mandel]

Aaahhh.. missed that.

So if I'm using SolrJ, do I need to set that explicitly through set(); or
can I use setFacetSort() somehow? ('cause I can't find an example anywhere,
and it's not inherently obvious).
http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/SolrQuery.html#setFacetSort(java.lang.String)

Much appreciated for pointing me in the right direction though.

Mark

On Mon, Apr 18, 2011 at 3:34 PM, Yonik Seeley wrote:

> On Mon, Apr 18, 2011 at 1:10 AM, Mark Mandel 
> wrote:
> > Is it possible to sort some facets by Lex, and others by numeric value?'
>
> Yes, see
>
> http://wiki.apache.org/solr/SimpleFacetParameters#Parameters
>
> Example of per-field overrides:
>
> facet.field=foo
> facet.field=bar
> f.foo.facet.sort=count
> f.bar.facet.sort=index
>
> -Yonik
> http://www.lucenerevolution.org -- Lucene/Solr User Conference, May
> 25-26, San Francisco
>



-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: *:* query with dismax

[Mark Mandel]

This is exactly what should be happening, as the dismax parser doesn't
understand regular query syntax (and for good reason too). This tripped me
up as well when I first started using dismax.

Solution for me was to comfigure the handler to use *:* when the query is
empty, so that you can still get back a full result set if you need it, say
for faceting.

HTH

Mark
On May 7, 2011 9:22 AM, "Jason Chaffee"  wrote:
> I am using dismax and trying to use q=*:* to return all indexed
> documents. However, it is always returning 0 found.
>
>
>
> If I used the default select (not dismax) handler and try q=*:* then it
> returns all documents.
>
>
>
> There is nothing in the logs to indicate why this happening.
>
>
>
> Does anyone have any clues?
>
>
>
> Thanks,
>
>
>
> Jason
>

Tokenising based on known words?

[Mark Mandel]

Not sure if this possible, but figured I would ask the question.

Basically, we have some users who do some pretty rediculous things ;o)

Rather than writing "red jacket", they write "redjacket", which obviously
returns no results.

Is there any way, with Solr, to go hunting for known words (maybe if there
is no results) within the word set? Or even tokenise based on known words in
the index?

Last time I played with spell check suggestions, it didn't seem to handle
this very well,  but I've yet to try it again on 3.2.0 (just upgraded from
1.4.1).

Any help/thoughts appreciated, as they do this al the time.

Mark

-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Tokenising based on known words?

[Mark Mandel]

Synonyms really wouldn't work for every possible combination of words in our
index.

Thanks for the idea though.

Mark

On Thu, Jun 9, 2011 at 3:42 PM, Gora Mohanty  wrote:

> On Thu, Jun 9, 2011 at 4:37 AM, Mark Mandel  wrote:
> > Not sure if this possible, but figured I would ask the question.
> >
> > Basically, we have some users who do some pretty rediculous things ;o)
> >
> > Rather than writing "red jacket", they write "redjacket", which obviously
> > returns no results.
> [...]
>
> Have you tried using synonyms,
>
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory
> It seems like they should fit your use case.
>
> Regards,
> Gora
>



-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Tokenising based on known words?

[Mark Mandel]

Thanks for the feedback! This definitely gives me some options to work on!

Mark

On Thu, Jun 9, 2011 at 11:21 PM, Steven A Rowe  wrote:

> Hi Mark,
>
> Are you familiar with shingles aka token n-grams?
>
>
> http://lucene.apache.org/solr/api/org/apache/solr/analysis/ShingleFilterFactory.html
>
> Use the empty string for the tokenSeparator to get wordstogether style
> tokens in your index.
>
> I think you'll want to apply this filter only at index-time, since the
> users will supply the shingles all by themselves :).
>
> Steve
>
> > -Original Message-
> > From: Mark Mandel [mailto:mark.man...@gmail.com]
> > Sent: Thursday, June 09, 2011 8:37 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Tokenising based on known words?
> >
> > Synonyms really wouldn't work for every possible combination of words in
> > our
> > index.
> >
> > Thanks for the idea though.
> >
> > Mark
> >
> > On Thu, Jun 9, 2011 at 3:42 PM, Gora Mohanty  wrote:
> >
> > > On Thu, Jun 9, 2011 at 4:37 AM, Mark Mandel 
> > wrote:
> > > > Not sure if this possible, but figured I would ask the question.
> > > >
> > > > Basically, we have some users who do some pretty rediculous things
> > ;o)
> > > >
> > > > Rather than writing "red jacket", they write "redjacket", which
> > obviously
> > > > returns no results.
> > > [...]
> > >
> > > Have you tried using synonyms,
> > >
> > >
> >
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymF
> > ilterFactory
> > > It seems like they should fit your use case.
> > >
> > > Regards,
> > > Gora
> > >
> >
> >
> >
> > --
> > E: mark.man...@gmail.com
> > T: http://www.twitter.com/neurotic
> > W: www.compoundtheory.com
> >
> > cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
> > http://www.cfobjective.com.au
> >
> > Hands-on ColdFusion ORM Training
> > www.ColdFusionOrmTraining.com
>



-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Hands-on ColdFusion ORM Training
www.ColdFusionOrmTraining.com

Re: Announcement/Invitation: Melbourne Solr/Lucene Users Group

[Mark Mandel]

Sounds great :) I'll sign up as well.

Look forward to a meeting!

Mark

On Thu, Jul 21, 2011 at 2:14 PM, Dave Hall  wrote:

> Hi Tal,
>
>
> On 21/07/11 14:04, Tal Rotbart wrote:
>
>> We have noticed that there is a blossoming of Apache Solr/Lucene usage
>> &  development in Melbourne in addition to a lack of an unofficial,
>> relaxed gathering to allow some fruitful information and experience
>> exchange.
>>
>> We're trying to put together a laid back meet up for developers (and
>> other interested people) who are currently using Apache Solr (and/or
>> Lucene) or would like to learn more about it.  Aiming for it to be a
>> high signal/noise ratio group, with meet ups probably once every two
>> months.
>>
>
> This sounds great!  I'm not sure I'll be a regular, but if I'm around town
> when it is on I will try to drop in.
>
>
>  The first meet up is still TBD, but please join the group if you're
>> keen to join us for pizza, beer, and a discussion about Solr once we
>> figure out the date of the first meeting.
>>
> Once a date is decided please update the Melbourne *UG wiki page so others
> can find out about it.  The wiki has meeting times for various user groups
> around town, which might help you find a time which doesn't clash with other
> groups.  Check out at 
> http://perl.net.au/wiki/**Melbourne
>
> Cheers
>
> Dave
>



-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) + Flex - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Re: Solr can not index "F**K"!

[Mark Mandel]

I hate to be the PC guy, but seriously, did this have to be said?

On Mon, Aug 1, 2011 at 6:58 AM, randohi  wrote:

> One of our clients (a hot girl!)




-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

cf.Objective(ANZ) + Flex - Nov 17, 18 - Melbourne Australia
http://www.cfobjective.com.au

Re: Using a sum of fields in a filter query

[Mark Mandel]

Thanks!

The piece I was also missing as well was to add:


To my solrconfig.xml.

Once I did that, it all worked perfectly!

Much appreciated!

Mark



On Tue, Sep 4, 2012 at 5:25 PM, Rafał Kuć  wrote:

> Hello!
>
> Try something like
>
> fq={!frange l=0 u=100}sum(fielda, fieldb, fieldc)
>
> --
> Regards,
>  Rafał Kuć
>  Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch
>
> > Hey all,
>
> > I have a series of fields with numeric values in my solr index.
>
> > What I want to be able to do is the equivalent of something like below in
> > my filter query (fq) parameter:
>
> > sum(fielda, fieldb, fieldc):[0 to 100]
>
> > So the sum of the fields is between 0 and 100.
>
> > Is there some way to do this in SOLR on a FQ?  I've hunted high and low
> and
> > haven't been able to see the correct syntax for it.
>
> > If someone could point me in the right direction, I would greatly
> > appreciated it.
>
> > Thanks!
>
> > Mark
>
>


-- 
E: mark.man...@gmail.com
T: http://www.twitter.com/neurotic
W: www.compoundtheory.com

2 Devs from Down Under Podcast
http://www.2ddu.com/

Re: Delete all documents in the index

[Mark Mandel]

Thanks for posting this!

I ran into exactly this issue yesterday, and ended up felting the files to
get around it.

Mark

Sent from my mobile doohickey.
On Sep 6, 2012 4:13 AM, "Rohit Harchandani"  wrote:

> Thanks everyone. Adding the _version_ field in the schema worked.
> Deleting the data directory works for me, but was not sure why deleting
> using curl was not working.
>
> On Wed, Sep 5, 2012 at 1:49 PM, Michael Della Bitta <
> michael.della.bi...@appinions.com> wrote:
>
> > Rohit:
> >
> > If it's easy, the easiest thing to do is to turn off your servlet
> > container, rm -r * inside of the data directory, and then restart the
> > container.
> >
> > Michael Della Bitta
> >
> > 
> > Appinions | 18 East 41st St., Suite 1806 | New York, NY 10017
> > www.appinions.com
> > Where Influence Isn’t a Game
> >
> >
> > On Wed, Sep 5, 2012 at 12:56 PM, Jack Krupansky  >
> > wrote:
> > > Check to make sure that you are not stumbling into SOLR-3432:
> > "deleteByQuery
> > > silently ignored if updateLog is enabled, but {{_version_}} field does
> > not
> > > exist in schema".
> > >
> > > See:
> > > https://issues.apache.org/jira/browse/SOLR-3432
> > >
> > > This could happen if you kept the new 4.0 solrconfig.xml, but copied in
> > your
> > > pre-4.0 schema.xml.
> > >
> > > -- Jack Krupansky
> > >
> > > -Original Message- From: Rohit Harchandani
> > > Sent: Wednesday, September 05, 2012 12:48 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: Delete all documents in the index
> > >
> > >
> > > Hi,
> > > I am having difficulty deleting documents from the index using curl.
> The
> > > urls i tried were:
> > > curl "http://localhost:9020/solr/core1/update/?stream.body=
> > > *:*&commit=true"
> > > curl "http://localhost:9020/solr/core1/update/?commit=true"; -H
> > > "Content-Type: text/xml" --data-binary 'id:[* TO
> > > *]'
> > > curl "http://localhost:9020/solr/core1/update/?commit=true"; -H
> > > "Content-Type: text/xml" --data-binary
> > '*:*'
> > > I also tried:
> > > curl "
> > >
> >
> http://localhost:9020/solr/core1/update/?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E&commit=true
> > > "
> > > as suggested on some forums. I get a response with status=0 in all
> cases,
> > > but none of the above seem to work.
> > > When I run
> > > curl "http://localhost:9020/solr/core1/select?q=*:*&rows=0&wt=xml";
> > > I still get a value for "numFound".
> > >
> > > I am currently using solr 4.0 beta version.
> > >
> > > Thanks for your help in advance.
> > > Regards,
> > > Rohit
> >
>