It sounds like it might be more efficient to implement this at the
crawler level to short-circuit crawling whole sites. Baring that, a
separate database sounds more flexible.
Non-deletable docs doesn't sound like something that should be a
general feature.
However, one would probably be able to i
Hi-
We recrawl the same places and update blindly without checking if a document
is already in the index. We have a use case where we would like to delete
documents (porn) and have them stay deleted. To implement this use case now,
we would need to check the existence of the document and check
