Hey, Thanks for the info. Yes; this generally confirms the issue. I see some of your higher slab classes with "free_chunks 0", so if you're setting data that requires these chunks it could error out. The "stats items" confirms this since there are no actual items in those lower slab classes.
You're certainly right a workaround of making your items < 512k would also work; but in general if I have features it'd be nice if they worked well :) Please open an issue so we can improve things! I intended to lower the slab_chunk_max default from 512k to much lower, as that actually raises the memory efficiency by a bit (less gap at the higher classes). That may help here. The system should also try ejecting items from the highest LRU... I need to double check that it wasn't already intending to do that and failing. Might also be able to adjust the page mover but not sure. The page mover can probably be adjusted to attempt to keep one page in reserve, but I think the algorithm isn't expecting slabs with no items in it so I'd have to audit that too. If you're up for experiments it'd be interesting to know if setting "-o slab_chunk_max=32768" or 16k (probably not more than 64) makes things better or worse. Also, crud.. it's documented as kilobytes but that's not working somehow? aaahahah. I guess the big EXPERIMENTAL tag scared people off since that never got reported. I'm guessing most people have a mix of small to large items, but you only have large items and a relatively low memory limit, so this is why you're seeing it so easily. I think most people setting large items have like 30G+ of memory so you end up with more spread around. Thanks, -Dormando On Wed, 24 Aug 2022, Hayden wrote: > What you're saying makes sense, and I'm pretty sure it won't be too hard to > add some functionality to my writing code to break my large items up into > smaller parts that can each fit into a single chunk. That has the added > benefit that I won't have to bother increasing the max item size. > In the meantime, though, I reran my pipeline and captured the output of > stats, stats slabs, and stats items both when evicting normally and when > getting > spammed with the error. > > First, the output when I'm in the error state: > **** Output of stats > STAT pid 1 > STAT uptime 11727 > STAT time 1661406229 > STAT version b'1.6.14' > STAT libevent b'2.1.8-stable' > STAT pointer_size 64 > STAT rusage_user 2.93837 > STAT rusage_system 6.339015 > STAT max_connections 1024 > STAT curr_connections 2 > STAT total_connections 8230 > STAT rejected_connections 0 > STAT connection_structures 6 > STAT response_obj_oom 0 > STAT response_obj_count 1 > STAT response_obj_bytes 65536 > STAT read_buf_count 8 > STAT read_buf_bytes 131072 > STAT read_buf_bytes_free 49152 > STAT read_buf_oom 0 > STAT reserved_fds 20 > STAT cmd_get 0 > STAT cmd_set 12640 > STAT cmd_flush 0 > STAT cmd_touch 0 > STAT cmd_meta 0 > STAT get_hits 0 > STAT get_misses 0 > STAT get_expired 0 > STAT get_flushed 0 > STAT delete_misses 0 > STAT delete_hits 0 > STAT incr_misses 0 > STAT incr_hits 0 > STAT decr_misses 0 > STAT decr_hits 0 > STAT cas_misses 0 > STAT cas_hits 0 > STAT cas_badval 0 > STAT touch_hits 0 > STAT touch_misses 0 > STAT store_too_large 0 > STAT store_no_memory 0 > STAT auth_cmds 0 > STAT auth_errors 0 > STAT bytes_read 21755739959 > STAT bytes_written 330909 > STAT limit_maxbytes 5368709120 > STAT accepting_conns 1 > STAT listen_disabled_num 0 > STAT time_in_listen_disabled_us 0 > STAT threads 4 > STAT conn_yields 0 > STAT hash_power_level 16 > STAT hash_bytes 524288 > STAT hash_is_expanding False > STAT slab_reassign_rescues 0 > STAT slab_reassign_chunk_rescues 0 > STAT slab_reassign_evictions_nomem 0 > STAT slab_reassign_inline_reclaim 0 > STAT slab_reassign_busy_items 0 > STAT slab_reassign_busy_deletes 0 > STAT slab_reassign_running False > STAT slabs_moved 0 > STAT lru_crawler_running 0 > STAT lru_crawler_starts 20 > STAT lru_maintainer_juggles 71777 > STAT malloc_fails 0 > STAT log_worker_dropped 0 > STAT log_worker_written 0 > STAT log_watcher_skipped 0 > STAT log_watcher_sent 0 > STAT log_watchers 0 > STAT unexpected_napi_ids 0 > STAT round_robin_fallback 0 > STAT bytes 5241499325 > STAT curr_items 4211 > STAT total_items 12640 > STAT slab_global_page_pool 0 > STAT expired_unfetched 0 > STAT evicted_unfetched 8429 > STAT evicted_active 0 > STAT evictions 8429 > STAT reclaimed 0 > STAT crawler_reclaimed 0 > STAT crawler_items_checked 4212 > STAT lrutail_reflocked 0 > STAT moves_to_cold 11872 > STAT moves_to_warm 0 > STAT moves_within_lru 0 > STAT direct_reclaims 55559 > STAT lru_bumps_dropped 0 > END > **** Output of stats slabs > STAT 2:chunk_size 120 > STAT 2:chunks_per_page 8738 > STAT 2:total_pages 1 > STAT 2:total_chunks 8738 > STAT 2:used_chunks 4211 > STAT 2:free_chunks 4527 > STAT 2:free_chunks_end 0 > STAT 2:get_hits 0 > STAT 2:cmd_set 0 > STAT 2:delete_hits 0 > STAT 2:incr_hits 0 > STAT 2:decr_hits 0 > STAT 2:cas_hits 0 > STAT 2:cas_badval 0 > STAT 2:touch_hits 0 > STAT 30:chunk_size 66232 > STAT 30:chunks_per_page 15 > STAT 30:total_pages 1 > STAT 30:total_chunks 15 > STAT 30:used_chunks 3 > STAT 30:free_chunks 12 > STAT 30:free_chunks_end 0 > STAT 30:get_hits 0 > STAT 30:cmd_set 0 > STAT 30:delete_hits 0 > STAT 30:incr_hits 0 > STAT 30:decr_hits 0 > STAT 30:cas_hits 0 > STAT 30:cas_badval 0 > STAT 30:touch_hits 0 > STAT 31:chunk_size 82792 > STAT 31:chunks_per_page 12 > STAT 31:total_pages 1 > STAT 31:total_chunks 12 > STAT 31:used_chunks 6 > STAT 31:free_chunks 6 > STAT 31:free_chunks_end 0 > STAT 31:get_hits 0 > STAT 31:cmd_set 0 > STAT 31:delete_hits 0 > STAT 31:incr_hits 0 > STAT 31:decr_hits 0 > STAT 31:cas_hits 0 > STAT 31:cas_badval 0 > STAT 31:touch_hits 0 > STAT 32:chunk_size 103496 > STAT 32:chunks_per_page 10 > STAT 32:total_pages 19 > STAT 32:total_chunks 190 > STAT 32:used_chunks 183 > STAT 32:free_chunks 7 > STAT 32:free_chunks_end 0 > STAT 32:get_hits 0 > STAT 32:cmd_set 0 > STAT 32:delete_hits 0 > STAT 32:incr_hits 0 > STAT 32:decr_hits 0 > STAT 32:cas_hits 0 > STAT 32:cas_badval 0 > STAT 32:touch_hits 0 > STAT 33:chunk_size 129376 > STAT 33:chunks_per_page 8 > STAT 33:total_pages 50 > STAT 33:total_chunks 400 > STAT 33:used_chunks 393 > STAT 33:free_chunks 7 > STAT 33:free_chunks_end 0 > STAT 33:get_hits 0 > STAT 33:cmd_set 0 > STAT 33:delete_hits 0 > STAT 33:incr_hits 0 > STAT 33:decr_hits 0 > STAT 33:cas_hits 0 > STAT 33:cas_badval 0 > STAT 33:touch_hits 0 > STAT 34:chunk_size 161720 > STAT 34:chunks_per_page 6 > STAT 34:total_pages 41 > STAT 34:total_chunks 246 > STAT 34:used_chunks 245 > STAT 34:free_chunks 1 > STAT 34:free_chunks_end 0 > STAT 34:get_hits 0 > STAT 34:cmd_set 0 > STAT 34:delete_hits 0 > STAT 34:incr_hits 0 > STAT 34:decr_hits 0 > STAT 34:cas_hits 0 > STAT 34:cas_badval 0 > STAT 34:touch_hits 0 > STAT 35:chunk_size 202152 > STAT 35:chunks_per_page 5 > STAT 35:total_pages 231 > STAT 35:total_chunks 1155 > STAT 35:used_chunks 1155 > STAT 35:free_chunks 0 > STAT 35:free_chunks_end 0 > STAT 35:get_hits 0 > STAT 35:cmd_set 0 > STAT 35:delete_hits 0 > STAT 35:incr_hits 0 > STAT 35:decr_hits 0 > STAT 35:cas_hits 0 > STAT 35:cas_badval 0 > STAT 35:touch_hits 0 > STAT 36:chunk_size 252696 > STAT 36:chunks_per_page 4 > STAT 36:total_pages 536 > STAT 36:total_chunks 2144 > STAT 36:used_chunks 2144 > STAT 36:free_chunks 0 > STAT 36:free_chunks_end 0 > STAT 36:get_hits 0 > STAT 36:cmd_set 0 > STAT 36:delete_hits 0 > STAT 36:incr_hits 0 > STAT 36:decr_hits 0 > STAT 36:cas_hits 0 > STAT 36:cas_badval 0 > STAT 36:touch_hits 0 > STAT 37:chunk_size 315872 > STAT 37:chunks_per_page 3 > STAT 37:total_pages 28 > STAT 37:total_chunks 84 > STAT 37:used_chunks 82 > STAT 37:free_chunks 2 > STAT 37:free_chunks_end 0 > STAT 37:get_hits 0 > STAT 37:cmd_set 0 > STAT 37:delete_hits 0 > STAT 37:incr_hits 0 > STAT 37:decr_hits 0 > STAT 37:cas_hits 0 > STAT 37:cas_badval 0 > STAT 37:touch_hits 0 > STAT 39:chunk_size 524288 > STAT 39:chunks_per_page 2 > STAT 39:total_pages 4212 > STAT 39:total_chunks 8424 > STAT 39:used_chunks 8422 > STAT 39:free_chunks 2 > STAT 39:free_chunks_end 0 > STAT 39:get_hits 0 > STAT 39:cmd_set 12640 > STAT 39:delete_hits 0 > STAT 39:incr_hits 0 > STAT 39:decr_hits 0 > STAT 39:cas_hits 0 > STAT 39:cas_badval 0 > STAT 39:touch_hits 0 > STAT active_slabs 10 > STAT total_malloced 5368709120 > END > **** Output of stats items > STAT items:39:number 4211 > STAT items:39:number_hot 768 > STAT items:39:number_warm 0 > STAT items:39:number_cold 3443 > STAT items:39:age_hot 28 > STAT items:39:age_warm 0 > STAT items:39:age 143 > STAT items:39:mem_requested 5241499325 > STAT items:39:evicted 8429 > STAT items:39:evicted_nonzero 0 > STAT items:39:evicted_time 140 > STAT items:39:outofmemory 0 > STAT items:39:tailrepairs 0 > STAT items:39:reclaimed 0 > STAT items:39:expired_unfetched 0 > STAT items:39:evicted_unfetched 8429 > STAT items:39:evicted_active 0 > STAT items:39:crawler_reclaimed 0 > STAT items:39:crawler_items_checked 4212 > STAT items:39:lrutail_reflocked 0 > STAT items:39:moves_to_cold 11872 > STAT items:39:moves_to_warm 0 > STAT items:39:moves_within_lru 0 > STAT items:39:direct_reclaims 8429 > STAT items:39:hits_to_hot 0 > STAT items:39:hits_to_warm 0 > STAT items:39:hits_to_cold 0 > STAT items:39:hits_to_temp 0 > END > > Then, the output when it's humming along happily again: > **** Output of stats > STAT pid 1 > STAT uptime 11754 > STAT time 1661406256 > STAT version b'1.6.14' > STAT libevent b'2.1.8-stable' > STAT pointer_size 64 > STAT rusage_user 3.056135 > STAT rusage_system 7.074541 > STAT max_connections 1024 > STAT curr_connections 3 > STAT total_connections 10150 > STAT rejected_connections 0 > STAT connection_structures 6 > STAT response_obj_oom 0 > STAT response_obj_count 1 > STAT response_obj_bytes 65536 > STAT read_buf_count 8 > STAT read_buf_bytes 131072 > STAT read_buf_bytes_free 49152 > STAT read_buf_oom 0 > STAT reserved_fds 20 > STAT cmd_get 0 > STAT cmd_set 12794 > STAT cmd_flush 0 > STAT cmd_touch 0 > STAT cmd_meta 0 > STAT get_hits 0 > STAT get_misses 0 > STAT get_expired 0 > STAT get_flushed 0 > STAT delete_misses 0 > STAT delete_hits 0 > STAT incr_misses 0 > STAT incr_hits 0 > STAT decr_misses 0 > STAT decr_hits 0 > STAT cas_misses 0 > STAT cas_hits 0 > STAT cas_badval 0 > STAT touch_hits 0 > STAT touch_misses 0 > STAT store_too_large 0 > STAT store_no_memory 0 > STAT auth_cmds 0 > STAT auth_errors 0 > STAT bytes_read 24375641173 > STAT bytes_written 415262 > STAT limit_maxbytes 5368709120 > STAT accepting_conns 1 > STAT listen_disabled_num 0 > STAT time_in_listen_disabled_us 0 > STAT threads 4 > STAT conn_yields 0 > STAT hash_power_level 16 > STAT hash_bytes 524288 > STAT hash_is_expanding False > STAT slab_reassign_rescues 0 > STAT slab_reassign_chunk_rescues 0 > STAT slab_reassign_evictions_nomem 0 > STAT slab_reassign_inline_reclaim 0 > STAT slab_reassign_busy_items 0 > STAT slab_reassign_busy_deletes 0 > STAT slab_reassign_running False > STAT slabs_moved 0 > STAT lru_crawler_running 0 > STAT lru_crawler_starts 20 > STAT lru_maintainer_juggles 71952 > STAT malloc_fails 0 > STAT log_worker_dropped 0 > STAT log_worker_written 0 > STAT log_watcher_skipped 0 > STAT log_watcher_sent 0 > STAT log_watchers 0 > STAT unexpected_napi_ids 0 > STAT round_robin_fallback 0 > STAT bytes 5242957328 > STAT curr_items 4212 > STAT total_items 12794 > STAT slab_global_page_pool 0 > STAT expired_unfetched 0 > STAT evicted_unfetched 8582 > STAT evicted_active 0 > STAT evictions 8582 > STAT reclaimed 0 > STAT crawler_reclaimed 0 > STAT crawler_items_checked 4212 > STAT lrutail_reflocked 0 > STAT moves_to_cold 12533 > STAT moves_to_warm 0 > STAT moves_within_lru 0 > STAT direct_reclaims 74822 > STAT lru_bumps_dropped 0 > END > **** Output of stats slabs > STAT 2:chunk_size 120 > STAT 2:chunks_per_page 8738 > STAT 2:total_pages 1 > STAT 2:total_chunks 8738 > STAT 2:used_chunks 4212 > STAT 2:free_chunks 4526 > STAT 2:free_chunks_end 0 > STAT 2:get_hits 0 > STAT 2:cmd_set 0 > STAT 2:delete_hits 0 > STAT 2:incr_hits 0 > STAT 2:decr_hits 0 > STAT 2:cas_hits 0 > STAT 2:cas_badval 0 > STAT 2:touch_hits 0 > STAT 30:chunk_size 66232 > STAT 30:chunks_per_page 15 > STAT 30:total_pages 1 > STAT 30:total_chunks 15 > STAT 30:used_chunks 3 > STAT 30:free_chunks 12 > STAT 30:free_chunks_end 0 > STAT 30:get_hits 0 > STAT 30:cmd_set 0 > STAT 30:delete_hits 0 > STAT 30:incr_hits 0 > STAT 30:decr_hits 0 > STAT 30:cas_hits 0 > STAT 30:cas_badval 0 > STAT 30:touch_hits 0 > STAT 31:chunk_size 82792 > STAT 31:chunks_per_page 12 > STAT 31:total_pages 1 > STAT 31:total_chunks 12 > STAT 31:used_chunks 6 > STAT 31:free_chunks 6 > STAT 31:free_chunks_end 0 > STAT 31:get_hits 0 > STAT 31:cmd_set 0 > STAT 31:delete_hits 0 > STAT 31:incr_hits 0 > STAT 31:decr_hits 0 > STAT 31:cas_hits 0 > STAT 31:cas_badval 0 > STAT 31:touch_hits 0 > STAT 32:chunk_size 103496 > STAT 32:chunks_per_page 10 > STAT 32:total_pages 19 > STAT 32:total_chunks 190 > STAT 32:used_chunks 183 > STAT 32:free_chunks 7 > STAT 32:free_chunks_end 0 > STAT 32:get_hits 0 > STAT 32:cmd_set 0 > STAT 32:delete_hits 0 > STAT 32:incr_hits 0 > STAT 32:decr_hits 0 > STAT 32:cas_hits 0 > STAT 32:cas_badval 0 > STAT 32:touch_hits 0 > STAT 33:chunk_size 129376 > STAT 33:chunks_per_page 8 > STAT 33:total_pages 50 > STAT 33:total_chunks 400 > STAT 33:used_chunks 391 > STAT 33:free_chunks 9 > STAT 33:free_chunks_end 0 > STAT 33:get_hits 0 > STAT 33:cmd_set 0 > STAT 33:delete_hits 0 > STAT 33:incr_hits 0 > STAT 33:decr_hits 0 > STAT 33:cas_hits 0 > STAT 33:cas_badval 0 > STAT 33:touch_hits 0 > STAT 34:chunk_size 161720 > STAT 34:chunks_per_page 6 > STAT 34:total_pages 41 > STAT 34:total_chunks 246 > STAT 34:used_chunks 246 > STAT 34:free_chunks 0 > STAT 34:free_chunks_end 0 > STAT 34:get_hits 0 > STAT 34:cmd_set 0 > STAT 34:delete_hits 0 > STAT 34:incr_hits 0 > STAT 34:decr_hits 0 > STAT 34:cas_hits 0 > STAT 34:cas_badval 0 > STAT 34:touch_hits 0 > STAT 35:chunk_size 202152 > STAT 35:chunks_per_page 5 > STAT 35:total_pages 231 > STAT 35:total_chunks 1155 > STAT 35:used_chunks 1155 > STAT 35:free_chunks 0 > STAT 35:free_chunks_end 0 > STAT 35:get_hits 0 > STAT 35:cmd_set 0 > STAT 35:delete_hits 0 > STAT 35:incr_hits 0 > STAT 35:decr_hits 0 > STAT 35:cas_hits 0 > STAT 35:cas_badval 0 > STAT 35:touch_hits 0 > STAT 36:chunk_size 252696 > STAT 36:chunks_per_page 4 > STAT 36:total_pages 536 > STAT 36:total_chunks 2144 > STAT 36:used_chunks 2144 > STAT 36:free_chunks 0 > STAT 36:free_chunks_end 0 > STAT 36:get_hits 0 > STAT 36:cmd_set 0 > STAT 36:delete_hits 0 > STAT 36:incr_hits 0 > STAT 36:decr_hits 0 > STAT 36:cas_hits 0 > STAT 36:cas_badval 0 > STAT 36:touch_hits 0 > STAT 37:chunk_size 315872 > STAT 37:chunks_per_page 3 > STAT 37:total_pages 28 > STAT 37:total_chunks 84 > STAT 37:used_chunks 84 > STAT 37:free_chunks 0 > STAT 37:free_chunks_end 0 > STAT 37:get_hits 0 > STAT 37:cmd_set 0 > STAT 37:delete_hits 0 > STAT 37:incr_hits 0 > STAT 37:decr_hits 0 > STAT 37:cas_hits 0 > STAT 37:cas_badval 0 > STAT 37:touch_hits 0 > STAT 39:chunk_size 524288 > STAT 39:chunks_per_page 2 > STAT 39:total_pages 4212 > STAT 39:total_chunks 8424 > STAT 39:used_chunks 8424 > STAT 39:free_chunks 0 > STAT 39:free_chunks_end 0 > STAT 39:get_hits 0 > STAT 39:cmd_set 12794 > STAT 39:delete_hits 0 > STAT 39:incr_hits 0 > STAT 39:decr_hits 0 > STAT 39:cas_hits 0 > STAT 39:cas_badval 0 > STAT 39:touch_hits 0 > STAT active_slabs 10 > STAT total_malloced 5368709120 > END > **** Output of stats items > STAT items:39:number 4212 > STAT items:39:number_hot 261 > STAT items:39:number_warm 0 > STAT items:39:number_cold 3951 > STAT items:39:age_hot 33 > STAT items:39:age_warm 0 > STAT items:39:age 165 > STAT items:39:mem_requested 5242957328 > STAT items:39:evicted 8582 > STAT items:39:evicted_nonzero 0 > STAT items:39:evicted_time 165 > STAT items:39:outofmemory 0 > STAT items:39:tailrepairs 0 > STAT items:39:reclaimed 0 > STAT items:39:expired_unfetched 0 > STAT items:39:evicted_unfetched 8582 > STAT items:39:evicted_active 0 > STAT items:39:crawler_reclaimed 0 > STAT items:39:crawler_items_checked 4212 > STAT items:39:lrutail_reflocked 0 > STAT items:39:moves_to_cold 12533 > STAT items:39:moves_to_warm 0 > STAT items:39:moves_within_lru 0 > STAT items:39:direct_reclaims 8582 > STAT items:39:hits_to_hot 0 > STAT items:39:hits_to_warm 0 > STAT items:39:hits_to_cold 0 > STAT items:39:hits_to_temp 0 > END > > I'm happy to open an issue on GitHub if the stats confirm there actually is > something in the code that could be fixed. You can decide then how much > effort it's worth to fix it. If my workaround idea works, though, I'll just > put it in place and move on to the next thing. ;-) > On Wednesday, August 24, 2022 at 7:01:33 PM UTC-7 Dormando wrote: > To put a little more internal detail on this: > > - As a SET is being processed item chunks must be made available > - If it is chunked memory, it will be fetching these data chunks from > across different slab classes (ie: 512k + 512k + sized enough for > whatever's left over) > - That full chunked item gets put in the largest slab class > - If another SET comes along and it needs 512k + 512k + an 8k, it has to > look into the 8k slab class for an item to evict. > - Except there's no memory in the 8k class: it's all actually in the > largest class. > - So there's nothing to evict to free up memory > - So you get an error. > - The slab page mover can make this worse by not leaving enough reserved > memory in the lower slab classes. > > I wasn't sure how often this would happen in practice and fixed a few > edge > cases in the past. Though I always figured I would've revisited it years > ago, so sorry about the trouble. > > There are a few tuning options: > 1) more memory, lol. > 2) you can override slab_chunk_max to be much lower (like 8k or 16k), > which will make a lot more chunks but you won't realistically notice a > performance difference. This can reduce the number of total slab > classes, > making it easier for more "end cap" memory to be found. > 3) delete items as you use them so it doesn't have to evict. not the > best > option. > > There're code fixes I can try but I need to see what the exact symptom > is > first, which is why I ask for the stats stuff. > > On Wed, 24 Aug 2022, dormando wrote: > > > Hey, > > > > You're probably hitting an edge case in the "large item support". > > > > Basically to store values > 512k memcached internally splits them up > into > > chunks. When storing items memcached first allocates the item storage, > > then reads data from the client socket directly into the data storage. > > > > For chunked items it will be allocating chunks of memory as it reads > from > > the socket, which can lead to that (thankfully very specific) "during > > read" error. I've long suspected some edge cases but haven't revisited > > that code in ... a very long time. > > > > If you can grab snapshots of "stats items" and "stats slabs" when it's > > both evicting normally and when it's giving you errors, I might be > able to > > figure out what's causing it to bottom out and see if there's some > tuning > > to do. Normal "stats" output is also helpful. > > > > It kind of smells like some slab classes are running low on memory > > sometimes, and the items in them are being read for a long time... > but we > > have to see the data to be sure. > > > > If you're feeling brave you can try building the current "next" branch > > from github and try it out, as some fixes to the page mover went in > there. > > Those fixes may have caused too much memory to be moved away from a > slab > > class sometimes. > > > > Feel free to open an issue on github to track this if you'd like. > > > > have fun, > > -Dormando > > > > On Wed, 24 Aug 2022, Hayden wrote: > > > > > Hello, > > > I'm trying to use memcached for a use case I don't think is > outlandish, but it's not behaving the way I expect. I > > > wanted to sanity check what I'm doing to see if it should be > working but there's maybe something I've done wrong > > > with my configuration, or if my idea of how it's supposed to work > is wrong, or if there's a problem with > > > memcached itself. > > > > > > I'm using memcached as a temporary shared image store in a > distributed video processing application. At the front > > > of the pipeline is a process (actually all these processes are pods > in a kubernetes cluster, if it matters, and > > > memcached is running in the cluster as well) that consumes a video > stream over RTSP, saves each frame to > > > memcached, and outputs events to a message bus (kafka) with > metadata about each frame. At the end of the pipeline > > > is another process that consumes these metadata events, and when it > sees events it thinks are interesting it > > > retrieves the corresponding frame from memcached and adds the frame > to a web UI. The video is typically 30fps, so > > > there are about 30 set() operations each second, and since each > value is effectively an image the values are a > > > bit big (around 1MB... I upped the maximum value size in memcached > to 2MB to make sure they'd fit, and I haven't > > > had any problems with my writes being rejected because of size). > > > > > > The video stream is processed in real-time, and effectively > infinite, but the memory available to memcached > > > obviously isn't (I've configured it to use 5GB, FWIW). That's OK, > because the cache is only supposed to be > > > temporary storage. My expectation is that once the available memory > is filled up (which takes a few minutes), > > > then roughly speaking for every new frame added to memcached > another entry (ostensibly the oldest one) will be > > > evicted. If the consuming process at the end of the pipeline > doesn't get to a frame it wants before it gets > > > evicted that's OK. > > > > > > That's not what I'm seeing, though, or at least that's not all that > I'm seeing. There are lots of evictions > > > happening, but the process that's writing to memcached also goes > through periods where every set() operation is > > > rejected with an "Out of memory during read" error. It seems to > happen in bursts where for several seconds every > > > write encounters the error, then for several seconds the set() > calls work just fine (and presumably other keys > > > are being evicted), then the cycle repeats. It goes on this way for > as long as I let the process run. > > > > > > I'm using memcached v1.6.14, installed into my k8s cluster using > the bitnami helm chart v6.0.5. My reading and > > > writing applications are both using pymemcache v3.5.2 for their > access. > > > > > > Can anyone tell me if it seems like what I'm doing should work the > way I described, and where I should try > > > investigating to see what's going wrong? Or alternatively, why what > I'm trying to do shouldn't work the way I > > > expected it to, so I can figure out how to make my applications > behave differently? > > > > > > Thanks, > > > Hayden > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > > To unsubscribe from this group and stop receiving emails from it, > send an email to > > > [email protected]. > > > To view this discussion on the web visit > > > > https://groups.google.com/d/msgid/memcached/702cae66-3108-46de-bb48-38eb3e17a5b7n%40googlegroups.com. > > > > > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/2e8a8cd4-cc13-3e78-f76-772b92374a9b%40rydia.net. > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/3c08514a-f43f-45aa-b25b-87b431cb74aen%40googlegroups.com. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/76e3d381-1ba9-e7f2-d4b2-30d87e7cb7e%40rydia.net.
