Hello, see inline. Markus -----Original message----- > From:Jim Anderson <jjanderson52...@gmail.com> > Sent: Tuesday 2nd June 2020 19:59 > To: solr-user@lucene.apache.org > Subject: Re: Building a web based search engine > > Hi Markus, > > Thanks for your response. I appreciate you giving me the bullet list of > things to do. I can take that list and work from it and hopefully make > progress, but I don't think it will get me where I want to be - just a bit > closer. > > You say, "We have been building precisely that for over ten years now". Is > it in a document? I would like to read it.
No, i haven't written a book about it and don't intend to. > Some basic things I would like to know that should be documented: > > 1) Using nutch as the crawler, how do I run a nutch thread that crawls my > named URLs. You don't, but run Nutch as a separate process from the command line. Or when you have to deal with 50+ million records, you run it on Apache Hadoop. > 2) I will use nutch to visit websites and create documents in solr. How do > I verify that documents have been created in Solr via nutch? By searching for them using Solr, or retrieving them by URL, using Solr's simple HTTP API. You can use SolrJ, the Java client, too. > 3) Solr will store and index the documents. How do I verify the index? See 2. > 4) I assume I can run a tomcat server on my host and then provide a > localhost URI to my web browser. Tomcat will then forward the URI to my > application. My application will take a query and using a java API is will > pass the query to Solr. I would like to see an example of a java program > passing a query to Solr. See 3. Though i would recommend to use Solr's HTTP API, it is much easier to deal with. > 5) Solr will take the query, parse it and then locate appropriate documents > using the index. Is there a log in Solr showing what queries have been > parsed? Yes, see Solr's log directory. > 6) Solr will pass back the list of documents it has located. I have not > really looked at this issue yet, but it would be nice to have an example of > this. Search for a SolrJ tutorial, they are plentiful. Also check out Solr's own extensive manual, everything you need is there. > Jim > > > > On Tue, Jun 2, 2020 at 12:12 PM Markus Jelsma <markus.jel...@openindex.io> > wrote: > > > Hello, > > > > We have been building precisely that for over ten years now. The '10,000 > > foot level overview' is basically: > > > > * forget about Lucene for now, Solr uses it under the hood; > > * get Solr, and start it with the schema.xml file that comes with Nutch; > > * get Nutch, give it a set of domains or hosts to crawl and some URLs to > > start the crawl with and point the indexer towards the previously > > configured Solr; > > * put a proxy in front of Solr (we use Nginx), or skip this step if it is > > just an internal demo (do not expose Solr to the outside world); > > * make some basic JS tool that handles input and search result responses. > > > > This was our first web search engine prototype and it was set up in a few > > days. The chapter "How To Build A Web Based Search Engine With Solr, Lucene > > and Nutch" just means: set up Solr, and point Nutch towards it, and tell it > > to start crawling and indexing. > > > > Then there comes and endless list of things to improve, autocomplete, > > spell checking, query and click log handling and analysis, proper text > > extraction, etc. > > > > Regards, > > Markus > > > > -----Original message----- > > > From:Jim Anderson <jjanderson52...@gmail.com> > > > Sent: Tuesday 2nd June 2020 16:36 > > > To: solr-user@lucene.apache.org > > > Subject: Building a web based search engine > > > > > > Hi, > > > > > > I have been looking at solr, lucene and nutch websites and tutuorials for > > > over a week now, experimenting and learning, but also frustrated be the > > > fact the I am totally missing the 'how to' do what I want. I see a lot of > > > examples of how to use each of the tools, but not how to put them all > > > together. I think an 'overview' at the 10,000 foot level is needed, Maybe > > > one is available and I have not yet found it. If someone can point me to > > > one, please do. > > > > > > If I am correct that an overview on "How To Build A Web Based Search > > Engine > > > With Solr, Lucene and Nutch" is not available, then I will be willing to > > > write an overview and make it available to the Solr community. I will > > need > > > input, explanation and review of others. > > > > > > My 2 goals are: > > > > > > 1) Build a demo web based search engine [Note: I have a very specific > > > business need to able to demonstrate a web application on top of a search > > > engine. This demo is intended to show a 'proof of concept' of the web > > > application to a small audience.] > > > > > > 2) Document the process of building the demo and customizing it using the > > > java API so that others can more easily build their own web base search > > > engine. > > > > > > Jim Anderson > > > > > >
