This is the second morning in a row I come to work to find our gitlab
server unwell. The symptoms are:
1. Going to the login page gives me a 500
2. Status seems OK, but stopping fails:
root@gitlab:/# gitlab-ctl status
run: gitlab-git-http-server: (pid 286) 82345s; down: log: 1s, normally up,
want up
run: logrotate: (pid 30879) 3123s; down: log: 1s, normally up, want up
run: nginx: (pid 308) 82343s; down: log: 1s, normally up, want up
run: postgresql: (pid 217) 82349s; down: log: 0s, normally up, want up
run: redis: (pid 209) 82351s; down: log: 1s, normally up, want up
run: sidekiq: (pid 276) 82346s; down: log: 0s, normally up, want up
run: unicorn: (pid 251) 82347s; down: log: 1s, normally up, want up
root@gitlab:/# gitlab-ctl stop
ok: down: gitlab-git-http-server: 0s, normally up
ok: down: logrotate: 0s, normally up
ok: down: nginx: 0s, normally up
ok: down: postgresql: 0s, normally up
timeout: run: redis: (pid 209) 82394s, want down, got TERM
timeout: run: sidekiq: (pid 276) 82420s, want down, got TERM
ok: down: unicorn: 0s, normally up
3. I see this repeating in the logs (when running `gitlab-ctl tail`):
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/ci/api/v1/builds/register.json" for 127.0.0.1 at 2015-11-04
08:10:22 +0000
Errno::EAGAIN (Resource temporarily unavailable - connect(2) for
/var/opt/gitlab/redis/redis.socket):
config/initializers/rack_attack_git_basic_auth.rb:5:in `block in <top
(required)>'
This is a description of my setup:
- GitLab is run in a Docker container, gitlab/gitlab-ce (124cd013a598).
- The container's `/etc/gitlab`, `/var/log/gitlab` and `/var/opt/gitlab`
are mapped onto folders on an NFS mounted area. (To ensure they get
backed up regularly.)
- I've configured GitLabCI and have one runner, it's running in a Docker
container on the same host, gitlab/gitlab-runner (d3b921e60fc1).
- I've configured LDAP authentication against our LDAP server.
- I've configured mail sending via GMail (I've not managed to verify
- that setup yet though).
I'm suspecting the use of NFS for the config/log/data is unusual, but
I've not seen anything mentioned about it. I've had this setup for a
few weeks now and not seen any problems with it until yesterday.
My current fix is to shut down gitlab, remove it, and then run a new
instance.
Any comments, suggestions, etc are welcome!
/M
--
Magnus Therning, [email protected]
Cipherstone Technologies AB
Theres Svenssons gata 10, 417 55 Gothenburg, Sweden
Time is a great teacher, but unfortunately it kills all its pupils.
-- Hector Louis Berlioz
--
You received this message because you are subscribed to the Google Groups
"GitLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/gitlabhq/20151104083838.GA4433%40sobel.cipherstone.com.
For more options, visit https://groups.google.com/d/optout.
