[Bug 1872118] Re: DHCP Cluster crashes after a few hours

** Changed in: bind9-libs (Ubuntu Focal) Status: New => In Progress ** Changed in: bind9-libs (Ubuntu Groovy) Status: New => In Progress ** Changed in: isc-dhcp (Ubuntu Focal) Status: New => In Progress ** Changed in: isc-dhcp (Ubuntu Groovy) Status: Confirmed => In P

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

just to clarify, the change is in bind9-libs and fixes the issue also for my servers. the change +--- bind9-libs-9.11.16+dfsg.orig/lib/isc/unix/socket.c bind9-libs-9.11.16+dfsg/lib/isc/unix/socket.c +@@ -4050,7 +4050,7 @@ check_write: + if (!SOCK_DEAD(sock)) { +

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Pretty sure its working now ;) been running 24 hours and over 2 million requests. Thanks for all the effort Jorge -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes a

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Since i replaced the bad link , the patch appears to be working. been running for 14 hours and over 1.2 Million requests both servers running fine. Many thanks for all your effots. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https:

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

No crashes to report. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manage notifications about this bug go to: https://bugs.launchpad.net/dh

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Hello @Andrew, @rlaager, Any crashes to report before I propose this patch? my env is running this patch for close to 3 days without any failures. Thanks, -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Jorge, it sounds like ISC might think there is a more fundamental issue here: https://gitlab.isc.org/isc-projects/dhcp/-/issues/121#note_152804 ** Bug watch added: gitlab.isc.org/isc-projects/dhcp/-/issues #121 https://gitlab.isc.org/isc-projects/dhcp/-/issues/121 -- You received this bug not

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

looks like it may be fixed , been running about 5 hours about 400k request and still alive. early signs are good, with the patch and my dodgy link removed. Will leave running over night -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. h

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I think on of the links was wrong, just remade it. and going to test again -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manage notification

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Jorge, Here you go andrew@gw2-focal:~$ ldd /usr/sbin/dhcpd | grep export libirs-export.so.161 => /lib/x86_64-linux-gnu/libirs-export.so.161 (0x00 007fa030e7b000) libdns-expo

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Hello Andrew, I just reviewed the core file that you provided. Thread 1 is the thread that panics on assertion because sock.pending_send is already set. This is the condition I prevented in the PPA, so *shouldn't* be hitting the frame 3 In my test systems I don't hit this condition, dispatch_se

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Not sure if this helps i coped the coredump file to a test system install gdb and got the following # gdb /usr/sbin/dhcpd ./CoreDump GNU gdb (Ubuntu 9.1-0ubuntu1) 9.1 Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Yes i restarted isc-dhcp-server with systemctl restart isc-dhcp-server ( even tried a reboot once) # dpkg -l | grep ppa1 ii isc-dhcp-server 4.4.1-2.1ubuntu6~ppa1 amd64ISC DHCP server for automatic IP address assignment ii libdns-export1109

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Jorge, I have been running for 25 hours on the patched version with no crashes on either server. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

OK, I have no crashes to report for the last 24 hours with the PPA included here. ● isc-dhcp-server.service - ISC DHCP IPv4 server Loaded: loaded (/lib/systemd/system/isc-dhcp-server.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-08-04 14:58:11 UTC; 1

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Running over a day with the packages from the PPA in place: ● isc-dhcp-server.service - ISC DHCP IPv4 server Loaded: loaded (/lib/systemd/system/isc-dhcp-server.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-08-04 16:23:19 UTC; 1 day 1h ago Docs:

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Last time in only did sudo apt install \ isc-dhcp-server=4.4.1-2.1ubuntu6~ppa1 \ sudo systemctl restart isc-dhcp-server This time i ried as someone else had done. sudo apt install \ isc-dhcp-server=4.4.1-2.1ubuntu6~ppa1 \ libdns-export1109=1:9.11.16+dfsg-3~ppa1 \ libirs-export16

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I ran: sudo apt install \ isc-dhcp-server=4.4.1-2.1ubuntu6~ppa1 \ libdns-export1109=1:9.11.16+dfsg-3~ppa1 \ libirs-export161=1:9.11.16+dfsg-3~ppa1 \ libisc-export1105=1:9.11.16+dfsg-3~ppa1 && \ sudo systemctl restart isc-dhcp-server The restart at the end was just for extra

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Andrew, Thank you for your input. ** Do you have any logs or a crash report to take a look after you upgraded these systems? On my test lab , I am counting for 3+ hours without a crash. root@dhcpd1:/home/ubuntu# dpkg -l | grep ppa1 ii isc-dhcp-server4.4.1-2.1ubuntu6~ppa1

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

My config files are PRIMARY SERVER CONFIG default-lease-time 600; max-lease-time 7200; failover peer "failover-partner" { primary; address 192.168.0.2; port 519; peer address 192.168.0.3; peer port 521; max-response-delay 60; max-unacked-

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

isc-dhcp-server failed to start after that, a reboot fixed it and it appeared to stabilise. Then as a test i upgraded the primary server. when the primary server came on line the secondary went off line and the stability of the environment went down fast. Remove the patched version and installed

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Hello Andrew, Correct me if I am wrong but it seems your system isn't running with libisc-export1105:amd64 1:9.11.16+dfsg-3~ppa1 (?) I am running the following packages from the PPA, please note that libisc-export1105 is required (that's where the fix is located). root@dhcpd1:/home/ubuntu# dpkg

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I've done the apt install on one of the failover server (Its the one that mostly crashes) Listing... Done isc-dhcp-server/focal,now 4.4.1-2.1ubuntu6~ppa1 amd64 [installed] still crashed syslog shows Aug 4 17:40:44 gw2-focal sh[74365]: ../../../../lib/isc/unix/socket.c:3361: INSIST(!sock->pend

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

error in the above description should have been I've done the apt install on one of SERVERS the failover server (Its the one that mostly crashes) Listing... Done isc-dhcp-server/focal,now 4.4.1-2.1ubuntu6~ppa1 amd64 [installed] still crashed syslog shows Aug 4 17:40:44 gw2-focal sh[74365]: .

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

also noticed this APPARMOR is not happy now Aug 4 09:37:37 gw2-focal kernel: [257023.493017] audit: type=1400 audit(1596530257.080:51): apparmor="DENIED" operation="open" profile="/usr/sbin/dhcpd" name="/proc/63448/task/63463/comm" pid=63448 comm="dhcpd" requested_mask="wr" denied_mask="wr" fs

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Hello Andrew, The fix is on the libisc-export1105 library (check dependencies on: https://launchpad.net/~niedbalski/+archive/ubuntu/fix-1872118/+packages), just replacing the dhcpd binary won't be enough. If you can install isc-dhcp-server and its dependencies from the PPA and test, would be gr

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

As my systems are pinned not to accept isc-dhcp-server upgrades, as I run the 18.04 version as it works. I extracted the dhcpd file from your packages and dropped it in place. It ran for about 5 hours with thousands of requests. Then it crashed with Aug 4 13:13:29 gw2-focal sh[63448]: ../../..

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

** Follow up from my previous comment, I've built a PPA with a fix similar to the one pointed in 430065. https://launchpad.net/~niedbalski/+archive/ubuntu/fix-1872118 * I'd appreciate if anyone can test that PPA with focal and report back if the problem keeps reproducible with that version. In

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Hello, I checked the backtrace of a crashed dhcpd running on 4.4.1-2.1ubuntu5. (gdb) info threads Id Target Id Frame * 1 Thread 0x7fb4ddecb700 (LWP 3170) __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 2 Thread 0x7fb4dd6ca700 (LWP 3171) __lll_lock_wait (futex=futex@ent

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

** Also affects: isc-dhcp (Ubuntu Groovy) Importance: Undecided Status: Confirmed ** Also affects: isc-dhcp (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: bind9-libs (Ubuntu) Importance: Undecided Status: New -- You received this bug notification

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Hello, I am trying to setup a reproducer for the mentioned issue. I have 2 machines acting as peers with the following versions: # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 20.04.1 LTS Release:20.04 Codename: focal # dpkg -l |grep -

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

** Changed in: dhcp Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manage notifications about this bug go to: h

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

** Bug watch added: gitlab.isc.org/isc-projects/dhcp/issues #128 https://gitlab.isc.org/isc-projects/dhcp/issues/128 ** Also affects: dhcp via https://gitlab.isc.org/isc-projects/dhcp/issues/128 Importance: Unknown Status: Unknown -- You received this bug notification because you

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I was able to reproduce this with 4.4.2 plus the Ubuntu packaging. I did not try with stock 4.4.2 from source. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I don't have specific recommendation on this. The best course of action would be to find a new maintainer that would update the packages to 4.4.2. The upstream (ISC) does very infrequent releases. We had 4.4.2 coming out in Jan 2020. We don't know when the next one (4.4.3) will come out, but we're

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Tomek, Please remember I tried the version 4.4.2 as suggested and as you say Ubuntu used 4.4.1, so it could be something to do with the 4.4.1 release or the ubuntu changes. Either way it looks like its not going to be fixed. Given this how do you apply to have a ISC-DHCP_Server removed form the Ubu

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Andrew, thanks for checking this up. Ok, so it looks like a problem with Ubuntu patches. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manag

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

tried it on a single server in the cluster and it looks like it better, i.e it actually launches, however there is no -user or -group options, with different directories and it must be started as root (with no change of id_ to start, so not secure. I guess thee other changes are added by ubuntu -

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Can someone affected by this bug compile 4.4.2 and see if this problem persists? You can get the latest sources from here: https://ftp.isc.org/isc/dhcp/4.4.2/ Compilation is trivial (untar, ./configure && make). The failover is enabled by default, so no need for extra switches to configure. If it

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Looking at the history around isc-dhcp, it appears if i read it properly that isc-dhcp sever for Ubuntu has been abandoned. If this is the case should it even be in the focal build -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

** Tags added: focal rls-ff-incoming -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manage notifications about this bug go to: https://bugs.l

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I just moved primary dns and dhcp from 16.04.6 LTS to 20.04 LTS and also observe this behavior. Primary is isc-dhcp-server 4.4.1-2.1ubuntu5 amd64. Crashes regularly. Secondary is isc-dhcp-server 4.3.5-3+deb9 armhf which runs fine. -- You received this bug notification because you are a member

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Thats what I did, but had a couple of outages, I had a choice of extending the lease time or downgrade. I did also try to disable apparmor, and dhcp gave an error (can't remember what though) and crashed I guess AppArmor just detected this and killed the process. So far about a week in ad every

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

I do have this problem running Ubuntu 20.04. Have been fiddling around with config, starting and stopping, apparmor and what not. Currently running it in single node with services stopped on the backup. Not very keen on downgrading but might have to go that path or migrate to KEA. A very big tha

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

No response, I actually wondering if I logged the bug in the wrong place. Anyone building a cluster of DHCP servers on Ubuntu 20.04 should run into this. The above or no one is using Ubuntu for DHCP clusters ? -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Any update on this? I just spent a few hours troubleshooting this only to find this bug. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manag

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

MM , Thanks I install the isc-dhcp-server from the Bionic-Security repository, and it appears to work. For anyone else here is how to install an older version of the server,and then block updates to the focal version PLEASE USE AT YOUR OWN RISK #Backup your old config sudo mkdir /etc/dhcp.20.04

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Thats not a bad idea, they It seems to be they are ignoring the critical issue. i.e you CAN@T RUN A PRODUCTION CLUSTER -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster cra

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

My current workaround: downgrade isc-dhcp-server to 18.04 package version. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To manage notification

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

each server starts if the other is not running, but starting them together caused this issue -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: DHCP Cluster crashes after a few hours To m

[Bug 1872118] Re: DHCP Cluster crashes after a few hours

Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: isc-dhcp (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1872118 Title: D