Hi All -
I'm seeing some strange behavior on a dev server. Right now the riak kv 2.2.0
server has 9GB free Ram, and no keys/values stored in Riak at all. (I've been
trying to figure out what's going on with the server, so I completely wiped
/var/lib/riak and re-installed from packagecould). Ulimit -n is set
appropriately as well.
If I make the following changes to /etc/riak/riak.conf I get crash error
messages:
storage_backend = leveldb
search = on
Turning search on with bitcask results in no such crash/error messages. When
switching between bitcask/leveldb on this test instance, i'm wiping the
/var/lib/riak directory each time.
Not much is turning up in google, and there is definitely plenty of memory
available for allocation.
Just wondering if anyone else has seen this.
Thanks!
The error messages:
(crash.log)
2017-01-18 15:27:23 =CRASH REPORT====
crasher:
initial call: yz_index_hashtree:init/1
pid: <0.2356.0>
registered_name: []
exception exit: {{{badmatch,{error,{db_open,"IO error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate
memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
ancestors: [yz_index_hashtree_sup,yz_general_sup,yz_sup,<0.711.0>]
messages: []
links: [<0.2206.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 1598
stack_size: 27
reductions: 689
neighbours:
2017-01-18 15:27:24 =SUPERVISOR REPORT====
Supervisor: {local,yz_index_hashtree_sup}
Context: child_terminated
Reason: {{badmatch,{error,{db_open,"IO error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate
memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
Offender:
[{pid,<0.2356.0>},{name,ignored},{mfargs,{yz_index_hashtree,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]
(console.log)
2017-01-18 15:35:39.100 [error] <0.3049.0> CRASH REPORT Process <0.3049.0> with
0 neighbours exited with reason: no match of right hand value
{error,{db_open,"IO error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in
gen_server:init_it/6 line 328
2017-01-18 15:35:39.101 [error] <0.2206.0> Supervisor yz_index_hashtree_sup had
child ignored started with {yz_index_hashtree,start_link,undefined} at
<0.3049.0> exit with reason no match of right hand value {error,{db_open,"IO
error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in context
child_terminated
2017-01-18 15:35:39.101 [info]
<0.3050.0>@riak_core_throttle:enable_throttle:119 Enabling throttle for
yokozuna/aae_throttle.
2017-01-18 15:35:39.101 [error] <0.3030.0> gen_server yz_entropy_mgr terminated
with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO
error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate
memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}}
in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371
2017-01-18 15:35:39.101 [error] <0.3030.0> CRASH REPORT Process yz_entropy_mgr
with 0 neighbours exited with reason: no match of right hand value
{error,{{badmatch,{error,{db_open,"IO error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate
memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}}
in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in
gen_server:terminate/6 line 744
2017-01-18 15:35:39.102 [error] <0.2010.0> Supervisor yz_general_sup had child
yz_entropy_mgr started with yz_entropy_mgr:start_link() at <0.3030.0> exit with
reason no match of right hand value {error,{{badmatch,{error,{db_open,"IO
error: lock
/var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK:
Cannot allocate
memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}}
in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in context
child_terminated
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
