Re: SpatialTierQueryParserPlugin Loading Error

[Estrada Groups]

Thanks a bunch for all the great responses! I think first thing tomorrow I will 
grab a fresh version from trunk then walk through the tutorial. I have not done 
that in quite some time...

I will also investigate the version in Git to see which one is easier to work 
with. I like the idea of building a JTS with a loose license which is what the 
advantage of the Apache license is, right? I suppose licensing is a whole other 
topic all together ;-)

Thanks again for all the great support!

Adam

On Dec 28, 2010, at 9:29 PM, Yonik Seeley  wrote:

> On Tue, Dec 28, 2010 at 8:54 PM, Adam Estrada  wrote:
>> I would gladly update this page if I could just get it working.
>> http://wiki.apache.org/solr/SpatialSearch
> 
> Everything on that wiki page should work w/o patches on trunk.
> I just ran through all of the examples, and everything seemed to be
> working fine.
> 
> -Yonik
> http://www.lucidimaignation.com

Re: Sub query using SOLR?

[Estrada Groups]

I am +1 on the interest on how to do this!

Adam


On Jan 4, 2011, at 1:26 PM, bbarani  wrote:

> 
> Hi,
> 
> I am trying to use subquery in SOLR, is there a way to implement this using
> SOLR query syntax?
> 
> Something like
> 
> Related_id: IN query(field=ud, q=”type:IT AND manager_12:dave”)
> 
> The thing I really want is to use output of one query to be the input of
> another query. 
> 
> Not sure if it is possible to use the query() function (function query) for
> my case..
> 
> Just want to know if ther is a better approach...
> 
> Thanks,
> Barani
> -- 
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Sub-query-using-SOLR-tp2193251p2193251.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: [sqljdbc4.jar] Errors

[Estrada Groups]

I downloaded that driver today and will test it tomorrow. Thanks for the tip! 
Would you mind sending an XML code snippet if it's any different to load than 
the MS driver?

Thanks,

Adam 

Sent from my iPhone

On Jan 4, 2011, at 11:34 PM, Gora Mohanty  wrote:

> On Wed, Jan 5, 2011 at 5:01 AM, Adam Estrada
>  wrote:
>> Can anyone help me with the following error. I upgraded my database to SQL
>> Server 2008 SP2 and now I get the following error. It was working with SQL
>> Server 2005.
>> 
>> 
>> Caused by: java.lang.UnsupportedOperationException: Java Runtime Environment
>> (JR
>> E) version 1.6 is not supported by this driver. Use the sqljdbc4.jar class
>> libra
>> ry, which provides support for JDBC 4.0.
> [...]
> 
> You need to do pretty much what the error message says. The last
> time we looked at MS SQL Server, the Microsoft JDBC driver had
> both a sqljdbc.jar, and a sqljdbc4.jar. The former was for Java 1.5,
> and the latter for Java 1.6.
> 
> You might also consider moving to the open source jTDS driver
> ( http://jtds.sourceforge.net/ ). We moved to it in the advice of
> some people on this list, and it does seem to perform better, drop
> connections less often, etc.
> 
> Regards,
> Gora

Re: [Solr4.0] Release Date

[Estrada Groups]

Thanks Grant! Ridiculous question #2. What is the difference between 3.1 and 
4.0? I am very confused about this...

Adam

Sent from my iPhone

On Jan 6, 2011, at 8:15 AM, Grant Ingersoll  wrote:

> I think 3.1 will be next, but I don't see a date yet.  I would hope sometime 
> soon, but I haven't checked the outstanding issues recently.
> 
> 
> On Jan 5, 2011, at 11:36 PM, Adam Estrada wrote:
> 
>> I know this is a bit premature but does anyone have an anticipated release
>> date for Solr4.0? I looked at Jira and there are still quite a few
>> unresolved issues in there. Do we think it will be spring, summer or fall
>> for this release?
>> 
>> Thanks,
>> Adam
> 
> --
> Grant Ingersoll
> http://www.lucidimagination.com
> 

Re: [Example] Compound Queries

[Estrada Groups]

I am using Solr4.0 for my testing right now if that helps.

Adam



On Jan 11, 2011, at 10:46 PM, Adam Estrada  
wrote:

> All,
> 
> I have the following query which works just fine for querying a date range.
> Now I would like to add any kind of spatial query to the mix. Would someone
> be so kind as to help me out with an example spatial query that works in
> conjunction with my date range query?
> 
> http://localhost:8983/solr/select/?q=hurricane+AND+eventdate:[2006-01-21T00:00:000Z+TO+2007-01-21T00:00:000Z]&version=2.2&start=0&rows=10&indent=on
> 
> I think it's something like this but my results are a not correct
> 
> http://localhost:8983/solr/select/?q=hurricane+AND+eventdate:[2006-01-21T00:00:000Z+TO+2007-01-21T00:00:000Z]&sfield=store&pt=45.15,-93.85&sort=geodist()%20asc&version=2.2&start=0&rows=10&indent=on
> 
> Your feedback is greatly appreciated!
> Adam

Re: Multi-word exact keyword case-insensitive search suggestions

[Estrada Groups]

Ahhh...the fun of open source software ;-). Requires a ton of trial and error! 
I found what worked for me and figured it was worth passing it along. If you 
don't mind...when you sort everything out on your end, please post results for 
the rest of us to take a gander at. 

Cheers,
Adam

On Jan 13, 2011, at 9:08 PM, Chamnap Chhorn  wrote:

> Thanks for your reply. However, it doesn't work for my case at all. I think
> it's the problem with query parser or something else. It forces me to put
> double quote to the search query in order to get the results found.
> 
> "sim 010"
> "sim 010"
> +DisjunctionMaxQuery((keyphrase:sim 010)) ()
> +(keyphrase:sim 010) ()
> 
> smart mobile
> smart mobile
> 
> +((DisjunctionMaxQuery((keyphrase:smart))
> DisjunctionMaxQuery((keyphrase:mobile)))~2) ()
> 
> +(((keyphrase:smart) (keyphrase:mobile))~2)
> ()
> 
> The intent here is to do a full text search, part of that is to search
> keyword field, so I can't put quote to it.
> 
> On Thu, Jan 13, 2011 at 10:30 PM, Adam Estrada <
> estrada.adam.gro...@gmail.com> wrote:
> 
>> Hi,
>> 
>> the following seems to work pretty well.
>> 
>>   > positionIncrementGap="100">
>> 
>>   
>>   > maxShingleSize="4" outputUnigrams="true"
>> outputUnigramIfNoNgram="false" />
>> 
>>   
>> 
>>   
>>   > autoGeneratePhraseQueries="true">
>> 
>>   
>>   
>>   
>>   >   ignoreCase="true"
>>   words="stopwords.txt"
>>   enablePositionIncrements="true"
>>   />
>>   > generateWordParts="1" generateNumberParts="1" catenateWords="1"
>> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
>>   
>>   > protected="protwords.txt"/>
>>   
>> 
>> 
>>   
>>   > ignoreCase="true" expand="true"/>
>>   >   ignoreCase="true"
>>   words="stopwords.txt"
>>   enablePositionIncrements="true"
>>   />
>>   > generateWordParts="1" generateNumberParts="1" catenateWords="0"
>> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
>>   
>>   > protected="protwords.txt"/>
>>   
>> 
>>   
>> 
>>   
>>   
>>   
>>   
>>   
>>   
>> 
>> I ingest the source fields as text_ws (I know I've changed it a bit) and
>> then copy the field to text. This seems to do what you are asking for.
>> 
>> Adam
>> 
>> On Thu, Jan 13, 2011 at 12:05 AM, Chamnap Chhorn >> wrote:
>> 
>>> Hi all,
>>> 
>>> I'm just stuck with exact keyword for several days. Hope you guys could
>>> help
>>> me. Here is the scenario:
>>> 
>>>  1. It need to be matched with multi-word keyword and case insensitive
>>>  2. Partial word or single word matching with this field is not allowed
>>> 
>>> I want to know the field type definition for this field and sample solr
>>> query. I need to combine this search with my full text search which uses
>>> dismax query.
>>> 
>>> Thanks
>>> --
>>> Chhorn Chamnap
>>> http://chamnapchhorn.blogspot.com/
>>> 
>> 
> 
> 
> 
> -- 
> Chhorn Chamnap
> http://chamnapchhorn.blogspot.com/

Re: Is solr 4.0 ready for prime time? (or other ways to use geo distance in search)

[Estrada Groups]

I tried to build yeaterdays svn trunk of 4.0 and got massive failures... The 
Hudson zipped up version seems to work without any issues. Has anyone else seem 
this build issue on the Mac? I guess this also has to do with Grants recent 
poll...

Adam


On Jan 22, 2011, at 6:34 AM, Robert Muir  wrote:

> On Fri, Jan 21, 2011 at 11:53 PM, Lance Norskog  wrote:
>> The Solr 4 branch is nowhere near ready for prime time. For example,
>> within the past week code was added that forces you to completely
>> reindex all of the documents you had. Solr 4 is really the "trunk".
>> The low-level stuff is being massively changed to allow very big
>> performance improvements and new features.
> 
> Changing the index format is not a sign of instability, we did this to
> improve performance. So, changing the index format is in no way a bad
> sign, nor indicative of whether or not the trunk is good for
> production use.
> 
> You aren't forced to re-index all your documents if you are riding
> trunk -- its your decision to make that tradeoff when you type 'svn
> update'. If you want stability you can take a snapshot (e.g. nightly
> build), and just stick with it.

Re: DIH From various File system locations

[Estrada Groups]

I would just use Nutch and specify the -solr param on the command line. That 
will add the extracted content your instance of solr.

Adam

Sent from my iPhone

On Jan 25, 2011, at 5:29 AM, pankaj bhatt  wrote:

> Hi All,
> I need to index the documents presents in my file system at various
> locations (e.g. C:\docs , d:\docs ).
>Is there any way through which i can specify this in my DIH
> Configuration.
>Here is my configuration:-
> 
> 
>  processor="FileListEntityProcessor"
>fileName="docx$|doc$|pdf$|xls$|xlsx|html$|rtf$|txt$|zip$"
> *baseDir="G:\\Desktop\\"*
>recursive="false"
>rootEntity="true"
>transformer="DateFormatTransformer"
> onerror="continue">
> processor="org.apache.solr.handler.dataimport.TikaEntityProcessor"
> url="${sd.fileAbsolutePath}" format="text" dataSource="bin">
>  
>  
>  
>  
>
> 
>
>
>
>
> 
>  
> 
> / Pankaj Bhatt.

Re: [Failure] to start Solr 4.0

[Estrada Groups]

It would seem like several of the sun projects require the "ant dist" command 
in order for everything to get built properly. Solr and Uima being two that 
have been mentioned in the last two days. Is there any way that we can include 
these in the root build script to cut out all the extra steps? This would make 
sense to a lot of folks I think...

Adam

On Jan 28, 2011, at 5:25 PM, Adam Estrada  wrote:

> I found the problem...You HAVE to build the Solr directory using "ant 
> example" in order for the web application to start properly. Sorry to post so 
> many times.
> 
> Adam
> 
> On Jan 28, 2011, at 5:20 PM, Adam Estrada wrote:
> 
>> All,
>> 
>> I've checked out the latest code and built the root directory with ant 
>> compile and then I build the solr directory again using the ant dist command 
>> which gives me the lucene-libs directory and a couple others. Now Solr won't 
>> start.  What am I missing???  This is as far as it gets.
>> 
>> mini:example Adam$ java -jar start.jar 
>> 2011-01-28 17:14:23.402:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
>> 2011-01-28 17:14:23.605:INFO::jetty-6.1.26
>> 2011-01-28 17:14:23.638:INFO::Started SocketConnector@0.0.0.0:8983
>> 
>> What couple possibly be the problem?
>> 
>> Adam
> 

Re: get SOMETHING out of an index

[Estrada Groups]

It would be really helpful to send along your schem.XML file so we can see how 
you are indexing these points. Polygons and linestrings are not supported yet. 
Another good way to test is using the Solr/admin tool or hand jamming your 
params in manually. Type *:* as your query in the admin tool. And see what it 
returns. It should return all indexed fields and their values.

Keep in mind that your radius search as to be done on the field type 
solr.LatLong so check out the field called stores in the example config file.  
From there you cam start to build out the rest of your queries starting with 
{!type=geofilt} I have example code that I can send along tomorrow. 

For the Solr/Lucene contributors out there, was it the point of storing lats 
and longs in individual fields if they can't really be used for anything? I 
they can please gimme an example that uses solr.point type.

Adam 

Sent from my iPhone

On Jan 29, 2011, at 11:09 PM, Dennis Gearon  wrote:

> I indexed my whole database (only 52k records).
> 
> It has some geospatioal on it. I set the geospatial to 1000km radius to 
> centered 
> on the town where they all are, and NADA comes out.
> 
> How can I find out what's in the index and get at least ONE document out?
> 
> Dennis Gearon
> 
> 
> Signature Warning
> 
> It is always a good idea to learn from your own mistakes. It is usually a 
> better 
> idea to learn from others’ mistakes, so you do not have to make them 
> yourself. 
> from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> 
> 
> EARTH has a Right To Life,
> otherwise we all die.
> 

Re: Solr for noSQL

[Estrada Groups]

What are the advantages of using something like HBase over your standard Lucene 
index with Solr? It would seem to me like you'd be losing a lot of what Lucene 
has to offer!?!

Adam

On Jan 31, 2011, at 5:34 AM, Steven Noels  wrote:

> On Fri, Jan 28, 2011 at 1:30 AM, Jianbin Dai  wrote:
> 
>> Hi,
>> 
>> 
>> 
>> Do we have data import handler to fast read in data from noSQL database,
>> specifically, MongoDB I am thinking to use?
>> 
>> Or a more general question, how does Solr work with noSQL database?
>> 
> 
> 
> Can't say anything about MongoDB, but we have an integration of SOLR with
> HBase inside Lily - www.lilyproject.org. It indeed uses the 'normal' SOLR
> index update API rather than a DIH - as we had the need to have incremental
> updates. The Indexer component we wrote does mapping from Lily/HBase schema
> to SOLR, as we also felt the need that both schemas shouldn't necessarily be
> identical.
> 
> Steven.
> -- 
> Steven Noels
> http://outerthought.org/
> Scalable Smart Data
> Makers of Kauri, Daisy CMS and Lily

Re: SOLR 1.4 and Lucene 3.0.3 index problem

[Estrada Groups]

I have the exact opposite problem where Luke won't even load the index but Solr 
starts fine. I believe there are major differences between the two indexes that 
are causing all these issues.

Adam



On Feb 1, 2011, at 6:28 AM, Churchill Nanje Mambe 
 wrote:

> hi guys
> I have developed a java crawler and integrated the lucene 3.0.3 API into it
> so it creates a Lucene.
> now I wish to search this lucene index using solr, I tried to configure the
> solrconfig.xml and schema.xml, everything seems to be fine
> but then solr told me the index is corrupt but I use luke and I am able to
> browse the index and perform searches and other things on it
> can someone help me which solr can wrap around a lucene 3.0.3 index ??
> regards
> 
> Mambe Churchill Nanje
> 237 33011349,
> AfroVisioN Founder, President,CEO
> http://www.afrovisiongroup.com | http://mambenanje.blogspot.com
> skypeID: mambenanje
> www.twitter.com/mambenanje

Re: CUSTOM JSP FOR APACHE SOLR

[Estrada Groups]

Has anyone noticed the rails application that installs with Solr4.0? I am 
interested to hear some feedback on that one...

Adam


On Jan 31, 2011, at 4:25 PM, Paul Libbrecht  wrote:

> Tomas,
> 
> I also know velocity can be used and works well.
> I would be interested to a simpler way to have the objects of SOLR available 
> in a jsp than write a custom jsp processor as a request handler; indeed, this 
> seems to be the way solrj is expected to be used in the wiki page.
> 
> Actually I migrated to velocity (which I like less than jsp) just because I 
> did not find a response to this question.
> 
> paul
> 
> 
> Le 31 janv. 2011 à 21:53, Tomás Fernández Löbbe a écrit :
> 
>> Hi John, you can use whatever you want for building your application, using
>> Solr on the backend (JSP included). You should find all the information you
>> need on Solr's wiki page:
>> http://wiki.apache.org/solr/
>> 
>> including some client libraries to easy
>> integrate your application with Solr:
>> http://wiki.apache.org/solr/IntegratingSolr
>> 
>> for fast prototyping you could
>> use Velocity:
>> http://wiki.apache.org/solr/VelocityResponseWriter
>> 
>> Anyway, I recommend you
>> to start with Solr's tutorial:
>> http://lucene.apache.org/solr/tutorial.html
>> 
>> 
>> Good luck,
>> Tomás
>> 
>> 2011/1/31 JOHN JAIRO GÓMEZ LAVERDE 
>> 
>>> 
>>> 
>>> SOLR LUCENE
>>> DEVELOPERS
>>> 
>>> Hi i am new to solr and i like to make a custom search page for enterprise
>>> users
>>> in JSP that takes the results of Apache Solr.
>>> 
>>> - Where i can find some useful examples for that topic ?
>>> - Is JSP the correct approach to solve mi requirement ?
>>> - If not what is the best solution to build a customize search page for my
>>> users?
>>> 
>>> Thanks
>>> from South America
>>> 
>>> JOHN JAIRO GOMEZ LAVERDE
>>> Bogotá - Colombia
>>> 
> 

Re: geodist and spacial search

[Estrada Groups]

Use the {!geofilt} param like Grant suggested. IMO, it works the best 
especially on larger datasets. 

Adam

Sent from my iPhone

On Feb 4, 2011, at 10:56 PM, Bill Bell  wrote:

> Why not just:
> 
> q=*:*
> fq={!bbox}
> sfield=store
> pt=49.45031,11.077721
> d=40
> fl=store
> sort=geodist() asc
> 
> 
> http://localhost:8983/solr/select?q=*:*&sfield=store&pt=49.45031,11.077721&;
> d=40&fq={!bbox}&sort=geodist%28%29%20asc
> 
> That will sort, and filter up to 40km.
> 
> No need for the 
> 
> fq={!func}geodist()
> sfield=store
> pt=49.45031,11.077721
> 
> 
> Bill
> 
> 
> 
> 
> On 2/4/11 4:30 AM, "Eric Grobler"  wrote:
> 
>> Hi Grant,
>> 
>> Thanks for the tip
>> This seems to work:
>> 
>> q=*:*
>> fq={!func}geodist()
>> sfield=store
>> pt=49.45031,11.077721
>> 
>> fq={!bbox}
>> sfield=store
>> pt=49.45031,11.077721
>> d=40
>> 
>> fl=store
>> sort=geodist() asc
>> 
>> 
>> On Thu, Feb 3, 2011 at 7:46 PM, Grant Ingersoll 
>> wrote:
>> 
>>> Use a filter query?  See the {!geofilt} stuff on the wiki page.  That
>>> gives
>>> you your filter to restrict down your result set, then you can sort by
>>> exact
>>> distance to get your sort of just those docs that make it through the
>>> filter.
>>> 
>>> 
>>> On Feb 3, 2011, at 10:24 AM, Eric Grobler wrote:
>>> 
 Hi Erick,
 
 Thanks I saw that example, but I am trying to sort by distance AND
>>> specify
 the max distance in 1 query.
 
 The reason is:
 running bbox on 2 million documents with a 20km distance takes only
>>> 200ms.
 Sorting 2 million documents by distance takes over 1.5 seconds!
 
 So it will be much faster for solr to first filter the 20km documents
>>> and
 then to sort them.
 
 Regards
 Ericz
 
 On Thu, Feb 3, 2011 at 1:27 PM, Erick Erickson
>>> >>> wrote:
 
> Further down that very page ...
> 
> Here's an example of sorting by distance ascending:
> 
> -
> 
> ...&q=*:*&sfield=store&pt=45.15,-93.85&sort=geodist()
> asc<
> 
>>> 
>>> http://localhost:8983/solr/select?wt=json&indent=true&fl=name,store&q=*:*
>>> &sfield=store&pt=45.15,-93.85&sort=geodist()%20asc
>> 
> 
> 
> 
> 
> The key is just the &sort=geodist(), I'm pretty sure that's
>>> independent
>>> of
> the bbox, but
> I could be wrong.
> 
> Best
> Erick
> 
> On Wed, Feb 2, 2011 at 11:18 AM, Eric Grobler <
>>> impalah...@googlemail.com
>> wrote:
> 
>> Hi
>> 
>> In http://wiki.apache.org/solr/SpatialSearch
>> there is an example of a bbox filter and a geodist function.
>> 
>> Is it possible to do a bbox filter and sort by distance - combine
>>> the
> two?
>> 
>> Thanks
>> Ericz
>> 
> 
>>> 
>>> --
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>> 
>>> Search the Lucene ecosystem docs using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>> 
>>> 
> 
> 

Re: [WKT] Spatial Searching

[Estrada Groups]

How could i stub this out not being a java guy? What is needed in order to do 
this? 

Licensing is always going to be an issue with JTS which is why I am interested 
in the project SIS sitting in incubation right now. 

I willing to put forth the effort if I had a little direction from the peanut 
gallery ;-)

Adam


On Feb 9, 2011, at 7:03 AM, Grant Ingersoll  wrote:

> The show stopper for JTS is it's license, unfortunately.  Otherwise, I think 
> it would be done already!  We could, since it's LGPL, make it an optional 
> dependency, assuming someone can stub it out.
> 
> On Feb 8, 2011, at 11:18 PM, Adam Estrada wrote:
> 
>> I just came across a ~nudge post over in the SIS list on what the status is 
>> for that project. This got me looking more in to spatial mods with Solr4.0.  
>> I found this enhancement in Jira. 
>> https://issues.apache.org/jira/browse/SOLR-2155. In this issue, David 
>> mentions that he's already integrated JTS in to Solr4.0 for querying on 
>> polygons stored as WKT. 
>> 
>> It's relatively easy to get WKT strings in to Solr but does the Field type 
>> exist yet? Is there a patch or something that I can test out? 
>> 
>> Here's how I would do it using GDAL/OGR and the already existing csv update 
>> handler. http://www.gdal.org/ogr/drv_csv.html
>> 
>> ogr2ogr -f CSV output.csv input.shp -lco GEOMETRY=AS_WKT
>> This converts a shapefile to a csv with the geometries in tact in the form 
>> of WKT. You can then get the data in to Solr by running the following 
>> command.
>> curl 
>> "http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,attr1,attr2,attr3,geom&stream.file=C:\tmp\output.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8";
>> There are lots of flavors of geometries so I suspect that this will be a 
>> daunting task but because JTS recognizes each geometry type it should be 
>> possible to work with them. 
>> Does anyone know of a patch or even when this functionality might be 
>> included in to Solr4.0? I need to query for polygons ;-)
>> Thanks,
>> Adam
>> 
>> 
>> 
> 
> --
> Grant Ingersoll
> http://www.lucidimagination.com/
> 

Re: [WKT] Spatial Searching

[Estrada Groups]

Thought I would share this on web mapping...it's a great write up and something 
to consider when talking about working with spatial data.

http://www.tokumine.com/2010/09/20/gis-data-payload-sizes/

Adam


On Feb 9, 2011, at 7:03 AM, Grant Ingersoll  wrote:

> The show stopper for JTS is it's license, unfortunately.  Otherwise, I think 
> it would be done already!  We could, since it's LGPL, make it an optional 
> dependency, assuming someone can stub it out.
> 
> On Feb 8, 2011, at 11:18 PM, Adam Estrada wrote:
> 
>> I just came across a ~nudge post over in the SIS list on what the status is 
>> for that project. This got me looking more in to spatial mods with Solr4.0.  
>> I found this enhancement in Jira. 
>> https://issues.apache.org/jira/browse/SOLR-2155. In this issue, David 
>> mentions that he's already integrated JTS in to Solr4.0 for querying on 
>> polygons stored as WKT. 
>> 
>> It's relatively easy to get WKT strings in to Solr but does the Field type 
>> exist yet? Is there a patch or something that I can test out? 
>> 
>> Here's how I would do it using GDAL/OGR and the already existing csv update 
>> handler. http://www.gdal.org/ogr/drv_csv.html
>> 
>> ogr2ogr -f CSV output.csv input.shp -lco GEOMETRY=AS_WKT
>> This converts a shapefile to a csv with the geometries in tact in the form 
>> of WKT. You can then get the data in to Solr by running the following 
>> command.
>> curl 
>> "http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,attr1,attr2,attr3,geom&stream.file=C:\tmp\output.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8";
>> There are lots of flavors of geometries so I suspect that this will be a 
>> daunting task but because JTS recognizes each geometry type it should be 
>> possible to work with them. 
>> Does anyone know of a patch or even when this functionality might be 
>> included in to Solr4.0? I need to query for polygons ;-)
>> Thanks,
>> Adam
>> 
>> 
>> 
> 
> --
> Grant Ingersoll
> http://www.lucidimagination.com/
> 

Re: How to use XML parser in DIH for a database?

[Estrada Groups]

If your using a DIH for SQL server you can configure it however you want. Here 
is a
snippet of my code. Note the Driver you need to grab from sourcenet.


 
 
   

 
 
 
 
 
 
 
 
 
 
 
 
 
> 
>driver="oracle.jdbc.driver.OracleDriver" 
>  url="jdbc:oracle:thin:@localhost:1521:xe" 
>  user="user" 
>  password="password"
>  name="ds"/>
>  
>  
> transformer="ClobTransformer">
>  
>   clob="true"/>
> processor="XPathEntityProcessor" forEach="/suppliers/supplier"
> dataField="clobxml.SUPPLIER_APPROVALS" onError="continue" >
>   />
>  
>
>
>  
> 
> 
> 
> -
> Thanx:
> Grijesh
> http://lucidimagination.com
> -- 
> View this message in context: 
> http://lucene.472066.n3.nabble.com/How-to-use-XML-parser-in-DIH-for-a-database-tp2508015p2515910.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to use XML parser in DIH for a database?

[Estrada Groups]

I was also gonna say why even worry about using XPath when you can write a SQL 
query to get your data out. Thats what i did and it seems much simpler and cuts 
out a step.

Adam

Sent from my iPhone

On Feb 16, 2011, at 10:21 PM, Bill Bell  wrote:

> Does anyone have an example of using this with SQL Server varchar or XML
> field?
> 
> ??
> 
> 
>
>
>
> forEach="/the/record/xpath" url="${y.xml_name}">
>
>
>
>
> 
> 
> 
> 
> On 2/16/11 2:17 AM, "Stefan Matheis"  wrote:
> 
>> What about using
>> http://wiki.apache.org/solr/DataImportHandler#XPathEntityProcessor ?
>> 
>> On Wed, Feb 16, 2011 at 10:08 AM, Bill Bell  wrote:
>>> I am using DIH.
>>> 
>>> I am trying to take a column in a SQL Server database that returns an
>>> XML
>>> string and use Xpath to get data out of it.
>>> 
>>> I noticed that Xpath works with external files, how do I get it to work
>>> with
>>> a database?
>>> 
>>> I need something like "//insur[5][@name='Blue Cross']"
>>> 
>>> Thanks.
>>> 
>>> 
>>> 
> 
> 

Re: DataImportHandler in Solr 4.0

[Estrada Groups]

Curious...why was this feature removed?

Adam

On Feb 23, 2011, at 6:55 PM, "Smiley, David W."  wrote:

> The DIH is no longer supplied embedded in the Solr war file.  You need to get 
> it on the classpath somehow. You could add another  solrconfig.xml to resolve this.
> 
> ~ David Smiley
> Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/
> 
> On Feb 23, 2011, at 4:11 PM, Alexandre Rocco wrote:
> 
>> Hi guys,
>> 
>> I'm having some issues when trying to use the DataImportHandler on Solr 4.0.
>> 
>> I've downloaded the latest nightly build of Solr 4.0 and configured normally
>> (on the example folder) solrconfig.xml file like this:
>> 
>> > class="org.apache.solr.handler.dataimport.DataImportHandler">
>> 
>> data-config.xml
>> 
>> 
>> 
>> At this point I noticed that the DIH jar was not being loaded correctly
>> causing exceptions like:
>> Error loading class 'org.apache.solr.handler.dataimport.DataImportHandler'
>> and
>> java.lang.ClassNotFoundException:
>> org.apache.solr.handler.dataimport.DataImportHandler
>> 
>> Do I need to build to get DIH running on Solr 4.0?
>> 
>> Thanks!
>> Alexandre
> 
> 
> 
> 
> 
> 
> 
> 

Re: Help please - recursively indexing lots and lots of text files

[Estrada Groups]

Nutch will also handle this but I'd probably stick with the DIH as Steve 
suggested. On windows it's pretty easy to get a list of all the txt file by 
using 

dir /b/s *.txt > files.txt

Just my $0.02 ;-)

Adam

Sent from my iPhone

On Mar 4, 2011, at 5:52 PM, Steven A Rowe  wrote:

> Hi Colin,
> 
> Solr's DataImportHandler sounds like what you want:
> 
>http://wiki.apache.org/solr/DataImportHandler
> 
> In particular, take a look at FileListEntityProcessor:
> 
>http://wiki.apache.org/solr/DataImportHandler#FileListEntityProcessor
> 
> Steve
> 
>> -Original Message-
>> From: csm [mailto:cmcswig...@gmail.com]
>> Sent: Friday, March 04, 2011 5:50 PM
>> To: solr-user@lucene.apache.org
>> Subject: Help please - recursively indexing lots and lots of text files
>> 
>> Hi,
>> 
>> I'm new to Lucene/Solr and I'm trying to build an index of a large body of
>> plaintext files for some corpus research that I'm doing.  There are about
>> 37,000 files of typically 50-100 lines each, and they're scattered
>> throughout a huge nested directory structure.  I've worked through the
>> basic
>> Solr tutorial and the text/html indexing tutorial at
>> http://www.slideshare.net/LucidImagination/indexing-text-and-html-files-
>> with-solr-4063407
>> , but after some looking around, I haven't been able to find any resources
>> for indexing a large number of text files that aren't all sitting in the
>> same directory.
>> 
>> Is this simply a case of having to write a shell script to crawl through
>> the
>> whole directory tree and call cURL for every single file, or is there a
>> library or utility that can do this, or just an easier way?  Any help
>> would
>> be greatly appreciated!  Alternatively, if this is a solved problem and I
>> just need to RTFM, it'd be great if someone could point me in the right
>> direction.
>> 
>> Thanks a lot,
>> Colin
>> 
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Help-
>> please-recursively-indexing-lots-and-lots-of-text-files-
>> tp2635884p2635884.html
>> Sent from the Solr - User mailing list archive at Nabble.com.

Re: [WKT] Spatial Searching

[Estrada Groups]

Outstanding! Thanks David...I can't wait to take a look at it.

Adam

Sent from my iPhone

On Mar 28, 2011, at 2:16 PM, "Smiley, David W."  wrote:

> (This is one of those messages that I would have responded to at the time if 
> I only noticed it.)
> 
> There is not yet indexing of arbitrary shapes (i.e. your data can only be 
> points), but with SOLR-2155 you can query via WKT thanks to JTS.  If you want 
> to index shapes then you'll have to wait a month or two for work that is 
> underway right now.  It's coming; be patient.
> 
> I don't see the LGPL licensing as a problem; it's *L*GPL, not GPL, after all. 
>  In SOLR-2155 the patch I take measures to download this library dynamically 
> at build time and compile against it.  JTS need not ship with Solr; the user 
> can get it themselves if they want this capability.  Non-JTS query shapes 
> should work without the presence of JTS.
> 
> ~ David Smiley
> Author: http://www.packtpub.com/solr-1-4-enterprise-search-server/
> 
> On Feb 8, 2011, at 11:18 PM, Adam Estrada wrote:
> 
>> I just came across a ~nudge post over in the SIS list on what the status is 
>> for that project. This got me looking more in to spatial mods with Solr4.0.  
>> I found this enhancement in Jira. 
>> https://issues.apache.org/jira/browse/SOLR-2155. In this issue, David 
>> mentions that he's already integrated JTS in to Solr4.0 for querying on 
>> polygons stored as WKT. 
>> 
>> It's relatively easy to get WKT strings in to Solr but does the Field type 
>> exist yet? Is there a patch or something that I can test out? 
>> 
>> Here's how I would do it using GDAL/OGR and the already existing csv update 
>> handler. http://www.gdal.org/ogr/drv_csv.html
>> 
>> ogr2ogr -f CSV output.csv input.shp -lco GEOMETRY=AS_WKT
>> This converts a shapefile to a csv with the geometries in tact in the form 
>> of WKT. You can then get the data in to Solr by running the following 
>> command.
>> curl 
>> "http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,attr1,attr2,attr3,geom&stream.file=C:\tmp\output.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8";
>> There are lots of flavors of geometries so I suspect that this will be a 
>> daunting task but because JTS recognizes each geometry type it should be 
>> possible to work with them. 
>> Does anyone know of a patch or even when this functionality might be 
>> included in to Solr4.0? I need to query for polygons ;-)
>> Thanks,
>> Adam

Re: Exporting to CSV

[Estrada Groups]

Check out the trunk version of Solr and build that. Those mods are in there for 
sure. I think the version in trunk is 4.0 but that discussion should be on a 
different thread ;-)

Adam


On Mar 29, 2011, at 11:35 PM, Charles Wardell  
wrote:

> Hi Koji,
> 
> Do you mean that adding &wt=csv to my http request will give me a csv?
> The only downloads that I see on the SOLR site is for 1.4.x
> Is there a 3.1 beta?
> 
> 
> On Mar 29, 2011, at 10:32 PM, Koji Sekiguchi wrote:
> 
>> (11/03/30 10:59), Charles Wardell wrote:
>>> Is there an easy way to get queried data exported from solr in a csv format?
>>> Hoping there is a handler or library for this.
>> 
>> Charlie,
>> 
>> Solr 3.1, will be released shortly, has csv response writer which is 
>> implicitly
>> defined. Try &wt=csv request parameter.
>> 
>> Koji
>> -- 
>> http://www.rondhuit.com/en/
> 

Indexing Flickr and Panaramio

[Estrada Groups]

Has anyone tried doing this? Got any tips for someone getting started?

Thanks,
Adam

Sent from my iPhone

Re: Indexing Flickr and Panaramio

[Estrada Groups]

Did this go to the list? I think I may need to resubscribe...

Sent from my iPhone

On Apr 12, 2011, at 12:55 AM, Estrada Groups  
wrote:

> Has anyone tried doing this? Got any tips for someone getting started?
> 
> Thanks,
> Adam
> 
> Sent from my iPhone

Re: Indexing Flickr and Panaramio

[Estrada Groups]

Thanks Peter! I am thinking that I may just use Nutch to do the crawl and index 
off of these sites. I need to check out the APIs for each to make sure I'm not 
missing anything related to the geospatial data for each image. Obviously both 
do the extraction when the images are uploaded so I'm guessing that it's also 
stored somewhere too ;-)

Adam 

Sent from my iPhone

On Apr 12, 2011, at 4:00 PM, Péter Király  wrote:

> Hi,
> 
> I did Flickr into Lucene about 3 years ago. There is a Flickr API,
> which covers almost everything you need (as I remember, not always
> Flickr feature was implemented at that time in the API, like the
> "collection" was not searchable). You can harvest by user ID or
> searching for a topic. You can use a language library (PHP, Java etc.)
> to wrap the details of communication. It is possible, that you would
> like to merge information into one entity before send to Solr (like
> merging the user, collection and set info into each pictures). The
> last step is to transform this information into a Solr document (again
> either directly or with a language library). I am not sure if it helps
> you, but if you ask more specific question, I try to answer.
> 
> regards,
> Péter
> 
> 2011/4/12 Estrada Groups :
>> Has anyone tried doing this? Got any tips for someone getting started?
>> 
>> Thanks,
>> Adam
>> 
>> Sent from my iPhone
>>