Resolved by removing /var/lib/gitlab/.gemhttps://github.com/rubygems/rubygems/issues/8639
Now gitlab always show 500 error and production.log has this (ruby-encryptor autopkgtest has been failing for some time)
OpenSSL::Cipher::CipherError (): encryptor (3.0.0) lib/encryptor.rb:98:in `final' encryptor (3.0.0) lib/encryptor.rb:98:in `crypt' encryptor (3.0.0) lib/encryptor.rb:49:in `decrypt' lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:16:in `decrypt_token' app/models/concerns/token_authenticatable_strategies/encrypted.rb:77:in `get_encrypted_token' app/models/concerns/token_authenticatable_strategies/encrypted.rb:112:in `token_set?' app/models/concerns/token_authenticatable_strategies/base.rb:46:in `ensure_token' app/models/concerns/token_authenticatable_strategies/encrypted.rb:37:in `ensure_token' app/models/concerns/token_authenticatable.rb:56:in `block in add_authentication_token_field' activesupport (7.0.8.6) lib/active_support/callbacks.rb:400:in `block in make_lambda' activesupport (7.0.8.6) lib/active_support/callbacks.rb:199:in `block (2 levels) in halting' activesupport (7.0.8.6) lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
activesupport (7.0.8.6) lib/active_support/callbacks.rb:686:in `catch'activesupport (7.0.8.6) lib/active_support/callbacks.rb:686:in `block in default_terminator' activesupport (7.0.8.6) lib/active_support/callbacks.rb:200:in `block in halting' activesupport (7.0.8.6) lib/active_support/callbacks.rb:595:in `block in invoke_before'
activesupport (7.0.8.6) lib/active_support/callbacks.rb:595:in `each'activesupport (7.0.8.6) lib/active_support/callbacks.rb:595:in `invoke_before' activesupport (7.0.8.6) lib/active_support/callbacks.rb:106:in `run_callbacks' activesupport (7.0.8.6) lib/active_support/callbacks.rb:929:in `_run_save_callbacks' activerecord (7.0.8.6) lib/active_record/callbacks.rb:455:in `create_or_update' activerecord (7.0.8.6) lib/active_record/timestamp.rb:126:in `create_or_update'
activerecord (7.0.8.6) lib/active_record/persistence.rb:615:in `save' activerecord (7.0.8.6) lib/active_record/validations.rb:47:in `save'activerecord (7.0.8.6) lib/active_record/transactions.rb:298:in `block in save' activerecord (7.0.8.6) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status' activerecord (7.0.8.6) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction' activesupport (7.0.8.6) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt' activesupport (7.0.8.6) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize' activesupport (7.0.8.6) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt' activesupport (7.0.8.6) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize' activerecord (7.0.8.6) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction' activerecord (7.0.8.6) lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer' lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write' lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'activerecord (7.0.8.6) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
activerecord (7.0.8.6) lib/active_record/transactions.rb:298:in `save' activerecord (7.0.8.6) lib/active_record/suppressor.rb:50:in `save' app/services/application_settings/update_service.rb:56:in `update_settings' app/services/application_settings/update_service.rb:12:in `execute' app/controllers/application_controller.rb:547:in `disable_usage_stats'app/controllers/application_controller.rb:534:in `set_usage_stats_consent_flag' activesupport (7.0.8.6) lib/active_support/callbacks.rb:400:in `block in make_lambda' activesupport (7.0.8.6) lib/active_support/callbacks.rb:199:in `block (2 levels) in halting' actionpack (7.0.8.6) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>' activesupport (7.0.8.6) lib/active_support/callbacks.rb:200:in `block in halting' activesupport (7.0.8.6) lib/active_support/callbacks.rb:595:in `block in invoke_before'
activesupport (7.0.8.6) lib/active_support/callbacks.rb:595:in `each'activesupport (7.0.8.6) lib/active_support/callbacks.rb:595:in `invoke_before' activesupport (7.0.8.6) lib/active_support/callbacks.rb:116:in `block in run_callbacks'
marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks' sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:32:in `block in sentry_around_action'
sentry-ruby (5.21.0) lib/sentry/hub.rb:108:in `with_child_span' sentry-ruby (5.21.0) lib/sentry-ruby.rb:499:in `with_child_span'sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action' activesupport (7.0.8.6) lib/active_support/callbacks.rb:127:in `block in run_callbacks' activesupport (7.0.8.6) lib/active_support/callbacks.rb:138:in `run_callbacks' actionpack (7.0.8.6) lib/abstract_controller/callbacks.rb:233:in `process_action' actionpack (7.0.8.6) lib/action_controller/metal/rescue.rb:23:in `process_action' actionpack (7.0.8.6) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action' activesupport (7.0.8.6) lib/active_support/notifications.rb:206:in `block in instrument' activesupport (7.0.8.6) lib/active_support/notifications/instrumenter.rb:24:in `instrument' activesupport (7.0.8.6) lib/active_support/notifications.rb:206:in `instrument' actionpack (7.0.8.6) lib/action_controller/metal/instrumentation.rb:66:in `process_action' actionpack (7.0.8.6) lib/action_controller/metal/params_wrapper.rb:259:in `process_action' activerecord (7.0.8.6) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (7.0.8.6) lib/abstract_controller/base.rb:151:in `process' actionview (7.0.8.6) lib/action_view/rendering.rb:39:in `process' actionpack (7.0.8.6) lib/action_controller/metal.rb:188:in `dispatch'actionpack (7.0.8.6) lib/action_controller/metal.rb:249:in `block in dispatch'
lib/gitlab/middleware/action_controller_static_context.rb:23:in `call' actionpack (7.0.8.6) lib/action_controller/metal.rb:249:in `dispatch'actionpack (7.0.8.6) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (7.0.8.6) lib/action_dispatch/routing/route_set.rb:32:in `serve'actionpack (7.0.8.6) lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
actionpack (7.0.8.6) lib/action_dispatch/routing/mapper.rb:48:in `serve'actionpack (7.0.8.6) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (7.0.8.6) lib/action_dispatch/journey/router.rb:32:in `each' actionpack (7.0.8.6) lib/action_dispatch/journey/router.rb:32:in `serve' actionpack (7.0.8.6) lib/action_dispatch/routing/route_set.rb:852:in `call' gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call' flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call' flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'
lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled' lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call' lib/gitlab/middleware/memory_report.rb:13:in `call' lib/gitlab/middleware/speedscope.rb:13:in `call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call' lib/gitlab/middleware/go.rb:21:in `call' lib/gitlab/etag_caching/middleware.rb:21:in `call' lib/gitlab/middleware/query_analyzer.rb:11:in `block in call' lib/gitlab/database/query_analyzer.rb:83:in `within' lib/gitlab/middleware/query_analyzer.rb:11:in `call' lib/ci/job_token/middleware.rb:11:in `call' batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call' rack-attack (6.7.0) lib/rack/attack.rb:103:in `call' /usr/lib/ruby/vendor_ruby/apollo_upload_server/middleware.rb:19:in `call' lib/gitlab/middleware/multipart.rb:173:in `call' rack-attack (6.7.0) lib/rack/attack.rb:127:in `call' warden (1.2.9) lib/warden/manager.rb:36:in `block in call' warden (1.2.9) lib/warden/manager.rb:34:in `catch' warden (1.2.9) lib/warden/manager.rb:34:in `call' /usr/lib/ruby/vendor_ruby/rack/cors.rb:102:in `call' rack (2.2.13) lib/rack/tempfile_reaper.rb:15:in `call' rack (2.2.13) lib/rack/etag.rb:27:in `call' rack (2.2.13) lib/rack/conditional_get.rb:40:in `call' rack (2.2.13) lib/rack/head.rb:12:in `call'actionpack (7.0.8.6) lib/action_dispatch/http/permissions_policy.rb:38:in `call' actionpack (7.0.8.6) lib/action_dispatch/http/content_security_policy.rb:36:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call' lib/gitlab/middleware/read_only.rb:18:in `call' lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call' rack (2.2.13) lib/rack/session/abstract/id.rb:266:in `context' rack (2.2.13) lib/rack/session/abstract/id.rb:260:in `call' actionpack (7.0.8.6) lib/action_dispatch/middleware/cookies.rb:704:in `call' lib/gitlab/middleware/same_site_cookies.rb:27:in `call'actionpack (7.0.8.6) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call' activesupport (7.0.8.6) lib/active_support/callbacks.rb:99:in `run_callbacks' actionpack (7.0.8.6) lib/action_dispatch/middleware/callbacks.rb:26:in `call' sentry-rails (5.21.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call' actionpack (7.0.8.6) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
lib/gitlab/middleware/path_traversal_check.rb:35:in `call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'
sentry-ruby (5.21.0) lib/sentry/hub.rb:265:in `with_session_tracking' sentry-ruby (5.21.0) lib/sentry-ruby.rb:412:in `with_session_tracking'sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'
sentry-ruby (5.21.0) lib/sentry/hub.rb:59:in `with_scope' sentry-ruby (5.21.0) lib/sentry-ruby.rb:392:in `with_scope' sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'actionpack (7.0.8.6) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call' /usr/lib/ruby/vendor_ruby/lograge/rails_ext/rack/logger.rb:15:in `call_app' railties (7.0.8.6) lib/rails/rack/logger.rb:25:in `block in call'activesupport (7.0.8.6) lib/active_support/tagged_logging.rb:99:in `block in tagged'
activesupport (7.0.8.6) lib/active_support/tagged_logging.rb:37:in `tagged' activesupport (7.0.8.6) lib/active_support/tagged_logging.rb:99:in `tagged' railties (7.0.8.6) lib/rails/rack/logger.rb:25:in `call'actionpack (7.0.8.6) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call' lib/gitlab/middleware/request_context.rb:15:in `call' lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call' /usr/lib/ruby/vendor_ruby/request_store/middleware.rb:19:in `call' rack (2.2.13) lib/rack/method_override.rb:24:in `call' rack (2.2.13) lib/rack/runtime.rb:22:in `call' rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call' rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout' rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call' config/initializers/fix_local_cache_middleware.rb:11:in `call' lib/gitlab/middleware/compressed_json.rb:44:in `call' actionpack (7.0.8.6) lib/action_dispatch/middleware/executor.rb:14:in `call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call' rack (2.2.13) lib/rack/sendfile.rb:110:in `call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call' gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call' gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context' gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call'actionpack (7.0.8.6) lib/action_dispatch/middleware/request_id.rb:26:in `call' actionpack (7.0.8.6) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
railties (7.0.8.6) lib/rails/engine.rb:530:in `call' railties (7.0.8.6) lib/rails/railtie.rb:226:in `public_send' railties (7.0.8.6) lib/rails/railtie.rb:226:in `method_missing' lib/gitlab/middleware/release_env.rb:12:in `call' rack (2.2.13) lib/rack/urlmap.rb:74:in `block in call' rack (2.2.13) lib/rack/urlmap.rb:58:in `each' rack (2.2.13) lib/rack/urlmap.rb:58:in `call' puma (6.4.3) lib/puma/configuration.rb:272:in `call' puma (6.4.3) lib/puma/request.rb:100:in `block in handle_request' puma (6.4.3) lib/puma/thread_pool.rb:378:in `with_force_shutdown' puma (6.4.3) lib/puma/request.rb:99:in `handle_request' puma (6.4.3) lib/puma/server.rb:464:in `process_client' puma (6.4.3) lib/puma/server.rb:245:in `block in run' puma (6.4.3) lib/puma/thread_pool.rb:155:in `block in spawn_thread'
OpenPGP_0x8F53E0193B294B75.asc
Description: OpenPGP public key
OpenPGP_signature.asc
Description: OpenPGP digital signature
