Many patterns against many sentences, storing all results

2016-01-05 Thread Will Moy
Hello Please may I have your advice as to whether Solr is a good tool for this job? We have (per year) – Up to 50,000,000 sentences And about 5,000 search patterns (i.e. queries) Our task is to identify all matches between any sentence and any search pattern. That list of detections must be kep

Re: Many patterns against many sentences, storing all results

2016-01-05 Thread Will Moy
Thank you both, that's really helpful. Luwak and Percolator look like good places to dig deeper. Best wishes Will *Will Moy* Director 020 3397 5140 *Full Fact* fullfact.org Twitter <https://twitter.com/FullFact> • Facebook <https://www.facebook.com/FullFact.org>

Solr filesystems: btrfs, xfs? Performance, stability, config...

2016-11-16 Thread Will Moy
Hi all Does anyone have any advice or experience on using btrfs or xfs for Solr? We've hit inode limits in ext (not because of Solr itself) and are wondering about using something else. I'm curious whether btrfs is stable enough, what configuration to use, whether one is better than the other, a