bin/post is not a crawler, just a small java class that collects links from html pages using SolrCell. It respects very basic robots.txt but far from the full spec. This is just a local prototyping tool, not meant for production use.
Jan Høydahl > 1. mar. 2020 kl. 09:27 skrev Mutuhprasannth <mprasanth...@gmail.com>: > > Have you found out the name of the crawler which is used by Solr bin/post or > how to ignore robots.txt in Solr post tool > > > > > -- > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
