Clashsoft commented on issue #5004: URL: https://github.com/apache/couchdb/issues/5004#issuecomment-2538986772
Now I'm stuck again with the original replication source, the npm registry: ``` [pod/couchdb-couchdb-2/couchdb] [notice] 2024-12-12T13:30:32.754275Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21036.0> -------- recording a checkpoint for `https://replicate.npmjs.com/registry/` -> `http://couchdb-svc-couchdb:5984/registry/` at source update_seq 2099 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.026452Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21036.0> -------- Replication `bc6b9cb696ae379a2a05de39631f5d0f+continuous` (`https://replicate.npmjs.com/registry/` -> `http://couchdb-svc-couchdb:5984/registry/`) failed: {bad_return_value,{invalid_json,{1,invalid_literal}}} [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027290Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21036.0> -------- gen_server <0.21036.0> terminated with reason: bad return value {invalid_json,{1,invalid_literal}} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{rep_id,{"bc6b9cb696ae379a2a05de39631f5d0f","+continuous"}},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{db_name,<<"shards/80000000-ffffffff/_replicator.1733607930">>},{doc_id,<<"3ddcb9c5f47447f9cae76d91e00022a1">>},{options,[{checkpoint_interval,5000},{connection_timeout,60000},{continuous,true},{create_target,false},{http_connections,20},{ibrowse_options,[]},{retries,5},{socket_options,[{keepalive,true},{nodelay,false}]},{use_bulk_get,false},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,1}]},{session_id,<<"dcd35a41280acb3dc66680605355d250">>},{start_seq,{0,0}},{source_seq,37198643},{committed_seq,{0,0}},{current_through_seq,{72,2099}},{highest_seq_done,{72,2099}}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027403Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21036.0> -------- gen_server <0.21036.0> terminated with reason: bad return value {invalid_json,{1,invalid_literal}} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{rep_id,{"bc6b9cb696ae379a2a05de39631f5d0f","+continuous"}},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{db_name,<<"shards/80000000-ffffffff/_replicator.1733607930">>},{doc_id,<<"3ddcb9c5f47447f9cae76d91e00022a1">>},{options,[{checkpoint_interval,5000},{connection_timeout,60000},{continuous,true},{create_target,false},{http_connections,20},{ibrowse_options,[]},{retries,5},{socket_options,[{keepalive,true},{nodelay,false}]},{use_bulk_get,false},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,1}]},{session_id,<<"dcd35a41280acb3dc66680605355d250">>},{start_seq,{0,0}},{source_seq,37198643},{committed_seq,{0,0}},{current_through_seq,{72,2099}},{highest_seq_done,{72,2099}}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027579Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21036.0> -------- CRASH REPORT Process (<0.21036.0>) with 3 neighbors exited with reason: bad return value {invalid_json,{1,invalid_literal}} at gen_server:handle_common_reply/8(line:1245) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: [couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 3, links: [<0.21162.0>,<0.21164.0>,<0.21165.0>,<0.21163.0>,<0.592.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,next => ...,...},...}},...], trap_exit: true, status: running, heap_size: 6772, stack_size: 28, reductions: 49346 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027674Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21036.0> -------- CRASH REPORT Process (<0.21036.0>) with 3 neighbors exited with reason: bad return value {invalid_json,{1,invalid_literal}} at gen_server:handle_common_reply/8(line:1245) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: [couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 3, links: [<0.21162.0>,<0.21164.0>,<0.21165.0>,<0.21163.0>,<0.592.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,next => ...,...},...}},...], trap_exit: true, status: running, heap_size: 6772, stack_size: 28, reductions: 49346 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027736Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21165.0> -------- gen_server <0.21165.0> terminated with reason: bad return value {invalid_json,{1,invalid_literal}} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{main_pid,<0.21036.0>},{loop,<0.21166.0>},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027777Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21165.0> -------- gen_server <0.21165.0> terminated with reason: bad return value {invalid_json,{1,invalid_literal}} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{main_pid,<0.21036.0>},{loop,<0.21166.0>},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027840Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21165.0> -------- CRASH REPORT Process (<0.21165.0>) with 0 neighbors exited with reason: bad return value {invalid_json,{1,invalid_literal}} at gen_server:decode_msg/9(line:909) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.21036.0>,couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 0, links: [<0.21166.0>], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 28, reductions: 6467 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:30:33.027924Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.21165.0> -------- CRASH REPORT Process (<0.21165.0>) with 0 neighbors exited with reason: bad return value {invalid_json,{1,invalid_literal}} at gen_server:decode_msg/9(line:909) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.21036.0>,couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 0, links: [<0.21166.0>], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 28, reductions: 6467 ``` And: ``` [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:37.673186Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25933.0> -------- Replicator, request POST to "https://replicate.npmjs.com/registry/_ensure_full_commit"; failed due to error {code,504} [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:37.673548Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25168.0> -------- Replication `bc6b9cb696ae379a2a05de39631f5d0f+continuous` (`https://replicate.npmjs.com/registry/` -> `http://couchdb-svc-couchdb:5984/registry/`) failed: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} [pod/couchdb-couchdb-0/couchdb] [notice] 2024-12-12T13:36:38.424668Z couc...@couchdb-couchdb-0.couchdb-couchdb.web-kraken.svc.cluster.local <0.18057.0> acfb8e39f5 couchdb.web-kraken.uni-kassel.dev 10.42.75.128 admin GET /_active_tasks 200 ok 3 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.048501Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25168.0> -------- gen_server <0.25168.0> terminated with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{rep_id,{"bc6b9cb696ae379a2a05de39631f5d0f","+continuous"}},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{db_name,<<"shards/80000000-ffffffff/_replicator.1733607930">>},{doc_id,<<"3ddcb9c5f47447f9cae76d91e00022a1">>},{options,[{checkpoint_interval,5000},{connection_timeout,60000},{continuous,true},{create_target,false},{http_connections,20},{ibrowse_options,[]},{retries,5},{socket_options,[{keepalive,true},{nodelay,false}]},{use_bulk_get,false},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,1}]},{session_id,<<"5bf633e8a5688c8804f002e40e2c48cc">>},{start_seq,{0,0}},{source_seq,37198716},{committed_seq,{0,0}},{current_through_seq,{2,13}},{highest_seq_done,{2,13}}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.048636Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25168.0> -------- gen_server <0.25168.0> terminated with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{rep_id,{"bc6b9cb696ae379a2a05de39631f5d0f","+continuous"}},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{db_name,<<"shards/80000000-ffffffff/_replicator.1733607930">>},{doc_id,<<"3ddcb9c5f47447f9cae76d91e00022a1">>},{options,[{checkpoint_interval,5000},{connection_timeout,60000},{continuous,true},{create_target,false},{http_connections,20},{ibrowse_options,[]},{retries,5},{socket_options,[{keepalive,true},{nodelay,false}]},{use_bulk_get,false},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,1}]},{session_id,<<"5bf633e8a5688c8804f002e40e2c48cc">>},{start_seq,{0,0}},{source_seq,37198716},{committed_seq,{0,0}},{current_through_seq,{2,13}},{highest_seq_done,{2,13}}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.050178Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25168.0> -------- CRASH REPORT Process (<0.25168.0>) with 3 neighbors exited with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} at gen_server:handle_common_reply/8(line:1241) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: [couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 3, links: [<0.25742.0>,<0.25744.0>,<0.25745.0>,<0.25743.0>,<0.592.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,next => ...,...},...}},...], trap_exit: true, status: running, heap_size: 6772, stack_size: 28, reductions: 29599 [pod/couchdb-couchdb-2/couchdb] [notice] 2024-12-12T13:36:44.050222Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.592.0> -------- couch_replicator_scheduler: Job {"bc6b9cb696ae379a2a05de39631f5d0f","+continuous"} started as <0.28499.0> [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.050353Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25168.0> -------- CRASH REPORT Process (<0.25168.0>) with 3 neighbors exited with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} at gen_server:handle_common_reply/8(line:1241) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors: [couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 3, links: [<0.25742.0>,<0.25744.0>,<0.25745.0>,<0.25743.0>,<0.592.0>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.34006561>,next => ...,...},...}},...], trap_exit: true, status: running, heap_size: 6772, stack_size: 28, reductions: 29599 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.050457Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25745.0> -------- gen_server <0.25745.0> terminated with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{main_pid,<0.25168.0>},{loop,<0.25746.0>},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.050565Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25745.0> -------- gen_server <0.25745.0> terminated with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} [pod/couchdb-couchdb-2/couchdb] last msg: redacted [pod/couchdb-couchdb-2/couchdb] state: [{main_pid,<0.25168.0>},{loop,<0.25746.0>},{source,"https://replicate.npmjs.com/registry/"},{target,"http://couchdb-svc-couchdb:5984/registry/"},{num_readers,0},{pending_fetch,nil},{batch_size,0}] [pod/couchdb-couchdb-2/couchdb] extra: [] [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.050673Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25745.0> -------- CRASH REPORT Process (<0.25745.0>) with 0 neighbors exited with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} at gen_server:decode_msg/9(line:909) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.25168.0>,couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 1, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 28, reductions: 915 [pod/couchdb-couchdb-2/couchdb] [error] 2024-12-12T13:36:44.050791Z couc...@couchdb-couchdb-2.couchdb-couchdb.web-kraken.svc.cluster.local <0.25745.0> -------- CRASH REPORT Process (<0.25745.0>) with 0 neighbors exited with reason: {checkpoint_commit_failure,<<"Failure on source commit: {'EXIT',{http_request_failed,\"POST\",\n \"https://replicate.npmjs.com/registry/_ensure_full_commit\",\n {error,{code,504}}}}">>} at gen_server:decode_msg/9(line:909) <= proc_lib:init_p_do_apply/3(line:240); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.25168.0>,couch_replicator_scheduler,couch_replicator_sup,...], message_queue_len: 1, links: [], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 28, reductions: 915 ``` I went with the config you suggested [here](https://github.com/apache/couchdb/issues/5004#issuecomment-2525305402) to track down the issue. But from the context of the log and the fact that not a single document was added, I suppose this time the problem might be the checkpoint? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
