RE: Installing Solr as a dependency
Thanks -- another interesting possibility, though I suppose the disadvantage to this strategy would be the dependency on Docker, which could be problematic for some users (especially those running Windows, where I understand that this could only be achieved with virtualization, which would almost certainly impact performance). Still, another option to put on the table! - Demian -Original Message- From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] Sent: Friday, July 29, 2016 8:02 PM To: solr-user Subject: Re: Installing Solr as a dependency What about (not tried) pulling down an official Docker build and adding your stuff to that? https://hub.docker.com/_/solr/ Regards, Alex. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com/ On 30 July 2016 at 03:03, Demian Katz wrote: >> I wouldn't include Solr in my own project at all. I would probably >> request that the user download the binary artifact and put it in a >> predictable location, and configure my installation script to do the >> download if the file is not there. I would strongly recommend taking >> advantage of Apache's mirror system for that download -- although if >> you need a specific version of Solr, you will find that the mirror >> system only has the latest version, and you must go to the Apache >> Archives for older versions. >> >> To reduce load on the Apache Archive, you could place a copy of the >> binary on your own download servers ... and you could probably >> greatly reduce the size of that download by stripping out components >> that your software doesn't need. If users want to enable additional >> functionality, they would be free to download the full Solr binary >> from Apache. > > Yes, this is the reason I was hoping to use some sort of dependency > management tool. The idea of downloading from Apache's system has definitely > crossed my mind, but it's inherently more fragile than using a dependency > manager (since Apache is at least theoretically free to change their URL > structure, etc., at any time) and, as you say, it seemed impolite to direct > potentially heavy amounts of traffic to Apache servers (especially when you > consider that every commit to my project triggers one or more continuous > integration builds, each of which would need to perform the download). > Creating a project-specific mirror also crossed my mind, but that has its own > set of problems: it's work to maintain it, and the server hosting it needs to > be able to withstand the high traffic that would otherwise be directed at > Apache. The idea of a theoretical dependency management tool still feels more > attractive because it adds a standard, unchanging mechanism for obtaining > specific versions of the software and it offers the possibility of local > package caching across builds to significantly reduce the amount of HTTP > traffic back and forth. Of course, it's a lot less attractive if it proves to > be only theory and not in fact practically achievable -- I'll play around > with Maven next week and see where that gets me. > > Anyway, I don't say any of that to dismiss your suggestions -- you > present potentially viable possibilities, and I'll certainly keep > those ideas on the table as I plan for the future -- but I thought it > might be worthwhile to share my thinking. :-) > >> I once discovered that if optional components are removed (including >> some jars in the webapp), the Solr download drops from 150+ MB to >> about >> 25 MB. >> >> https://issues.apache.org/jira/browse/SOLR-6806 > > This could actually be a separate argument for a dependency-management-based > Solr structure, in that you could create a core solr package with minimum > content that could recommend a whole array of optional dependencies. A script > could then be used to build different versions of the download package from > these -- one with just the core, one with all the optional stuff included. > Those who wanted some intermediate number of files could be encouraged to > manually create their desired build from packages. > > But again, I freely admit that everything I'm saying is based on > experience with package managers outside the realm of Java -- I need > to learn more about Maven (and perhaps Ivy) before I can make any > particularly intelligent statements about what is really possible in > this context. :-) > > - Demian
Question about Simple Post tool
Hi Guys, I have a quick question. I read the appropriate documentation and it seems that it is possible, but I might be getting the syntax wrong. I wish to use the simple Post Tool to pass in a URL that brings back a word document, and I Want to index the return of that url using TIka - Is that possible? Or do I have to get the file onto my file system first? Thanks, Sas
Re: Question about Simple Post tool
I don't think it's possible purely using the out-of-box post.jar. But why not disassemble post.jar (or get the source from internet) and modify it yourself. It seems not that hard. Scott Chu，scott@udngroup.com 2016/8/1 (週一) - Original Message - From: Jamal, Sarfaraz To: solr-user CC: Date: 2016/8/1 (週一) 22:05 Subject: Question about Simple Post tool Hi Guys, I have a quick question. I read the appropriate documentation and it seems that it is possible, but I might be getting the syntax wrong. I wish to use the simple Post Tool to pass in a URL that brings back a word document, and I Want to index the return of that url using TIka - Is that possible? Or do I have to get the file onto my file system first? Thanks, Sas - 未在此訊息中找到病毒。 已透過 AVG 檢查 - www.avg.com 版本: 2015.0.6201 / 病毒庫: 4627/12724 - 發佈日期: 08/01/16
RE: Question about Simple Post tool
Thank you. That is a great suggestion - Sas -Original Message- From: Scott Chu [mailto:scott@udngroup.com] Sent: Monday, August 1, 2016 10:21 AM To: solr-user Subject: Re: Question about Simple Post tool I don't think it's possible purely using the out-of-box post.jar. But why not disassemble post.jar (or get the source from internet) and modify it yourself. It seems not that hard. Scott Chu，scott@udngroup.com 2016/8/1 (週一) - Original Message - From: Jamal, Sarfaraz To: solr-user CC: Date: 2016/8/1 (週一) 22:05 Subject: Question about Simple Post tool Hi Guys, I have a quick question. I read the appropriate documentation and it seems that it is possible, but I might be getting the syntax wrong. I wish to use the simple Post Tool to pass in a URL that brings back a word document, and I Want to index the return of that url using TIka - Is that possible? Or do I have to get the file onto my file system first? Thanks, Sas - 未在此訊息中找到病毒。 已透過 AVG 檢查 - www.avg.com 版本: 2015.0.6201 / 病毒庫: 4627/12724 - 發佈日期: 08/01/16
Installing Solr with Ivy
As a follow-up to last week's thread about loading Solr via dependency manager,
I started experimenting with using Ivy to install Solr. Here's what I have
(note that I'm trying to install Solr 5.5.0 as an arbitrary example, but that
detail should not be important):
ivy.xml:
build.xml:
My hope, based on a quick read of some Ivy tutorials, was that simply running
"ant" with the above configs would give me a copy of Solr in my lib directory.
When I use example libraries from the tutorials in my ivy.xml, I do indeed get
files installed... but when I try to substitute the Solr package,
no files are installed ("0 artifacts copied"). I'm not very experienced with
any of these tools or repositories, so I'm not sure where I'm going wrong.
- Do I need to add some extra configuration somewhere to tell Ivy to download
the constituent parts of the solr-parent package?
- Is the solr-parent package the wrong thing to be using? (I tried replacing
solr-parent with solr-core and ended up with many .jar files in my lib
directory, which was better than nothing, but the .jar files were not organized
into a directory structure and were not accompanied by any of the non-.jar
files like shell scripts that make Solr tick).
- Am I just completely on the wrong track? (I do realize that there may not be
a way to pull a fully-functional Solr out of the core Maven repository... but
it seemed worth a try!)
Any suggestions would be greatly appreciated!
thanks,
Demian
Replication with managed resources?
I've a single core index with a managed schema, synonyms and stopwords that I'm thinking of making a master/slave pair via replication. How does does the replication confFiles option work with managed resources? Should I use their internal '_managed_xxx' filenames? -- View this message in context: http://lucene.472066.n3.nabble.com/Replication-with-managed-resources-tp4289880.html Sent from the Solr - User mailing list archive at Nabble.com.
How to set credentials when querying using SolrJ - Basic Authentication
Hello, I am looking to pass user / pwd when querying using CloudSolrClient. The documentation https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin describes about setting the credential when calling the request method like below SolrRequest req ;//create a new request object req.setBasicAuthCredentials(userName, password); solrClient.request(req); BUT how do we set the credentials when calling //HOW to set credentials before calling query method ??? ??? solrClient.query(collection, query);method? Looking the CloudSolrClient source code, i see query method creates a new QueryRequest object and thus doesn't provide a easy way to set credentials. Is there any way to easily hook/set credentials when calling query method using SolrJ? Thanks, Susheel ==
Re: Example of posting to /stream in SolrJ?
Sorry I was the one that initiated the cross-post ;-) Thanks for the pointer, works great! On Tue, Jul 26, 2016 at 3:32 PM, Joel Bernstein wrote: > I posted this also to another thread, but I'll cross post to this ticket: > > Take a look at org.apache.solr.client.solrj.io.sql. > StatementImpl.constructStream() > > This uses a SolrStream to connect to the /sql handler. You can use the same > approach to send a request to the /stream handler just by changing the > parameters. Then you can open and read the SolrStream. > > > > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Tue, Jul 26, 2016 at 3:58 PM, Timothy Potter > wrote: > >> Does anyone have an example of just POST'ing a streaming expression to >> the /stream handler from SolrJ client code? i.e. I don't want to parse >> and execute the streaming expression on the client side, rather, I >> want to post the expression to the server side. >> >> Currently, my client code is a big copy and paste of the /stream >> request handler, but I'd rather not do that. Specifically, I wasn't >> able to figure out how to parse the tuple >> stream coming back using SolrJ code if I just post the expression to >> /stream. >> >> Thanks. >>
Re: How to set credentials when querying using SolrJ - Basic Authentication
On 8/1/2016 1:59 PM, Susheel Kumar wrote:
> BUT how do we set the credentials when calling
>
> //HOW to set credentials before calling query method
> ???
> ???
> solrClient.query(collection, query);method?
Here's an example of setting credentials on an arbitrary request object
that is then sent to a specific collectionthat should always work:
SolrClient client = new CloudSolrClient("localhost:9893");
String collection = "gettingstarted";
String username = "test";
String password = "password";
SolrQuery query = new SolrQuery();
query.setQuery("*:*");
// Do any other query setup needed.
SolrRequest req = new QueryRequest(query);
req.setBasicAuthCredentials(username, password);
QueryResponse rsp = req.process(client, collection);
System.out.println("numFound: " + rsp.getResults().getNumFound());
You can use a similar approach with UpdateRequest to what I've done here
with QueryRequest.
When you use the sugar methods (like query, update, commit, etc),
SolrClient builds a request object and then uses the "process" method on
the request. The example above just makes this more explicit.
After I wrote the above code, I discovered that there is an alternate
request method that includes the collection parameter. This method
exists in the 5.4 version of SolrJ, which is the minimum version
required for setBasicAuthCredentials. I think the user code would be
about the same size either way.
Thanks,
Shawn
Re: Installing Solr with Ivy
On 8/1/2016 9:04 AM, Demian Katz wrote:
> As a follow-up to last week's thread about loading Solr via dependency
> manager, I started experimenting with using Ivy to install Solr. Here's what
> I have (note that I'm trying to install Solr 5.5.0 as an arbitrary example,
> but that detail should not be important):
> My hope, based on a quick read of some Ivy tutorials, was that simply running
> "ant" with the above configs would give me a copy of Solr in my lib
> directory. When I use example libraries from the tutorials in my ivy.xml, I
> do indeed get files installed... but when I try to substitute the Solr
> package,
> no files are installed ("0 artifacts copied"). I'm not very experienced with
> any of these tools or repositories, so I'm not sure where I'm going wrong.
>
> - Do I need to add some extra configuration somewhere to tell Ivy to download
> the constituent parts of the solr-parent package?
> - Is the solr-parent package the wrong thing to be using? (I tried replacing
> solr-parent with solr-core and ended up with many .jar files in my lib
> directory, which was better than nothing, but the .jar files were not
> organized into a directory structure and were not accompanied by any of the
> non-.jar files like shell scripts that make Solr tick).
> - Am I just completely on the wrong track? (I do realize that there may not
> be a way to pull a fully-functional Solr out of the core Maven repository...
> but it seemed worth a try!)
The general use for ivy is to download development libraries as part of
the build process. Downloading applications might be possible, but it's
a little outside what it was designed to do.
Looking into what's in solr-parent in maven central, it appears that the
only thing this contains is a Maven POM (an XML file) -- no binary
artifacts at all. I doubt that's useful. As you already noticed,
solr-core just gives you lots of jars that would let you embed a Solr
server into your own code -- it's not a full application.
In a theoretical situation where your program talked an SQL database,
would you include a database server in your project? How much time
would you invest in automating the download and install of MySQL,
Postgres, or some other database? I think what you would do in that
situation is include client code to talk to the database and expect the
user to provide the server and prepare it for your program. In this
respect, how is a Solr server any different than a database server?
Thanks,
Shawn
Re: How to set credentials when querying using SolrJ - Basic Authentication
Thank you so much, Shawn. Didn't realize that i could call process
directly. I think it will be helpful to add this code to solr
documentation. I'll create a jira to update the documentation.
Thanks,
Susheel
On Mon, Aug 1, 2016 at 7:14 PM, Shawn Heisey wrote:
> On 8/1/2016 1:59 PM, Susheel Kumar wrote:
> > BUT how do we set the credentials when calling
> >
> > //HOW to set credentials before calling query method
> > ???
> > ???
> > solrClient.query(collection, query);method?
>
> Here's an example of setting credentials on an arbitrary request object
> that is then sent to a specific collectionthat should always work:
>
> SolrClient client = new CloudSolrClient("localhost:9893");
> String collection = "gettingstarted";
> String username = "test";
> String password = "password";
>
> SolrQuery query = new SolrQuery();
> query.setQuery("*:*");
> // Do any other query setup needed.
>
> SolrRequest req = new QueryRequest(query);
> req.setBasicAuthCredentials(username, password);
> QueryResponse rsp = req.process(client, collection);
> System.out.println("numFound: " + rsp.getResults().getNumFound());
>
> You can use a similar approach with UpdateRequest to what I've done here
> with QueryRequest.
>
> When you use the sugar methods (like query, update, commit, etc),
> SolrClient builds a request object and then uses the "process" method on
> the request. The example above just makes this more explicit.
>
> After I wrote the above code, I discovered that there is an alternate
> request method that includes the collection parameter. This method
> exists in the 5.4 version of SolrJ, which is the minimum version
> required for setBasicAuthCredentials. I think the user code would be
> about the same size either way.
>
> Thanks,
> Shawn
>
>
Regarding HTMLStripCharFilter.
Hi,
Kindly help me understand the way HTMLStripCharFilter works.
I have following analysis chain.
int flags = WordDelimiterFilter.GENERATE_WORD_PARTS
| WordDelimiterFilter.GENERATE_NUMBER_PARTS
| WordDelimiterFilter.CATENATE_WORDS
| WordDelimiterFilter.CATENATE_NUMBERS
| WordDelimiterFilter.CATENATE_ALL
| WordDelimiterFilter.SPLIT_ON_CASE_CHANGE
| WordDelimiterFilter.STEM_ENGLISH_POSSESSIVE
| WordDelimiterFilter.PRESERVE_ORIGINAL;
@Override
protected Reader initReader(String field, Reader reader) {
return new HTMLStripCharFilter(reader);
}
@Override
protected TokenStreamComponents createComponents(String arg0) {
Tokenizer source = new WhitespaceTokenizer();
TokenStream wordDMTStrem = new WordDelimiterFilter(source, flags,
null);
TokenStream rdtStream = new
RemoveDuplicatesTokenFilter(wordDMTStrem);
return new TokenStreamComponents(source, rdtStream);
}
*teRm3* returns following analyzed tokens by above analysis
chain.
*Text Position IncrementPosition Length Offset attribute*
teRm3 11 0,
16
Rm3 11
0, 16
te 01
0, 16
teRm3 01 0,
16
Here in the above table teRm3 has occurred twice but not removed by
RemoveDuplicatesTokenFilter.
Whereas *teRm3* gets tokenized with the same analysis chain as below .
*Text Position IncrementPosition LengthOffset attribute*
teRm3 1 1 0, 5
te 0 1 0, 2
Rm3 1 1 2, 5
Here in above table *teRm3* was removed by RemoveDuplicatesTokenFilter so
no duplicate for it.
Please share your comments on this difference in behavior of analysis.
Thanks,
Modassar
solr error
Hi, i am connecting solr with php and getting *HTTP Error 52, and *HTTP Error 20* error *frequently . what should i do to minimize these issues . Regards, Abhishek T
Re: solr error
please reply . On Tue, Aug 2, 2016 at 10:24 AM, Midas A wrote: > Hi, > > i am connecting solr with php and getting *HTTP Error 52, and *HTTP Error > 20* error *frequently . > what should i do to minimize these issues . > > Regards, > Abhishek T > >
Re: solr error
I recommend you look at the PHP documentation to find out what “HTTP Error 52” means. You can start by searching the web for this: php http error 52 wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 1, 2016, at 10:04 PM, Midas A wrote: > > please reply . > > On Tue, Aug 2, 2016 at 10:24 AM, Midas A wrote: > >> Hi, >> >> i am connecting solr with php and getting *HTTP Error 52, and *HTTP Error >> 20* error *frequently . >> what should i do to minimize these issues . >> >> Regards, >> Abhishek T >> >>
Re: solr error
Abhishek, given the vast amount of information you write, I suspect thisis not an HTTP error code (those are three digits, and the ones starting with 200 actually indicate a success), but rather a libcurl error code. Check against this list to find out whether that's an explanation: https://curl.haxx.se/libcurl/c/libcurl-errors.html At least error 52 sounds familiar. Cheers, --Jürgen On 02.08.2016 07:04, Midas A wrote: > please reply . > > On Tue, Aug 2, 2016 at 10:24 AM, Midas A wrote: > >> Hi, >> >> i am connecting solr with php and getting *HTTP Error 52, and *HTTP Error >> 20* error *frequently . >> what should i do to minimize these issues . >> >> Regards, >> Abhishek T >> >>
Re: solr error
curl: (52) Empty reply from server what could be the case .and what should i do to minimize. On Tue, Aug 2, 2016 at 10:38 AM, Walter Underwood wrote: > I recommend you look at the PHP documentation to find out what “HTTP Error > 52” means. > > You can start by searching the web for this: php http error 52 > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > > On Aug 1, 2016, at 10:04 PM, Midas A wrote: > > > > please reply . > > > > On Tue, Aug 2, 2016 at 10:24 AM, Midas A wrote: > > > >> Hi, > >> > >> i am connecting solr with php and getting *HTTP Error 52, and *HTTP > Error > >> 20* error *frequently . > >> what should i do to minimize these issues . > >> > >> Regards, > >> Abhishek T > >> > >> > >
Re: solr error
Jürgen, we are using Php solrclient and getting above exception . what could be the reason for the same please elaborate. On Tue, Aug 2, 2016 at 11:10 AM, Midas A wrote: > curl: (52) Empty reply from server > what could be the case .and what should i do to minimize. > > > > > On Tue, Aug 2, 2016 at 10:38 AM, Walter Underwood > wrote: > >> I recommend you look at the PHP documentation to find out what “HTTP >> Error 52” means. >> >> You can start by searching the web for this: php http error 52 >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> >> > On Aug 1, 2016, at 10:04 PM, Midas A wrote: >> > >> > please reply . >> > >> > On Tue, Aug 2, 2016 at 10:24 AM, Midas A wrote: >> > >> >> Hi, >> >> >> >> i am connecting solr with php and getting *HTTP Error 52, and *HTTP >> Error >> >> 20* error *frequently . >> >> what should i do to minimize these issues . >> >> >> >> Regards, >> >> Abhishek T >> >> >> >> >> >> >
