I have an index of about 3,000,000 products and about 8500 customers. Each customers has access to about 50 to about 500,000 of the products.
Our current method was using a bitset in the filter. So, for each customer, they have a bitset in the cache. For each docId that they have access to, the bit is set. This is probably the best performance-wise for searches, but it consumes a lot of memory, especially because each document that they don't have access to also consumes space (a 0). It also is probably the cause of our problems when either these customer access lists (stored in files) or the index is updated. Is there a better way to manage access control? I was thinking of storing the user access list as a specific document type in the index. Basically, a single multi-value field. But I'm not quite sure where to go from here.
