Hello Please may I have your advice as to whether Solr is a good tool for this job?
We have (per year) – Up to 50,000,000 sentences And about 5,000 search patterns (i.e. queries) Our task is to identify all matches between any sentence and any search pattern. That list of detections must be kept up to date as patterns are added or updated (a handful an hour), and as new sentences are added. Some of the sentences will be added in real time, at probably max 100 / second and usually much less. The detections on these should be provided within 3 seconds. It's an unusual application in that we want all results in an external DB, and also in that every sentence is either a hit or not. we don't care about scoring results, only about matches for the exact search pattern entered. The application is automatically detecting instances of factchecked statements. The smaller-scale prototype was done with postgres full text searching, but that can't do exact phrase matching or other more sophisticated searches, so it's out. Thanks very much Will
