Hi Darren,
this is what Kea logs shortly before terminating the HA connection (any
personal information was removed from the logs):
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN
[kea-dhcp4.lease-cmds-hooks.139670034884352] LEASE_CMDS_UPDATE4_CONFLICT
lease4-update command failed due to conflict (parameters: { "client-id": "",
"expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev":
false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0,
"subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP
address:123.123.123.123 could not be updated.)
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN
[kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT
lease4-update command failed due to conflict (parameters: { "client-id": "",
"expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev":
false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0,
"subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP
address:123.123.123.123 could not be updated.)
Mar 12 19:27:28 dhcp-2 kea-dhcp4[203037]: WARN
[kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT
lease4-update command failed due to conflict (parameters: { "client-id": "",
"expire": 1678688848, "force-create": true, "fqdn-fwd": false, "fqdn-rev":
false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0,
"subnet-id": 3812, "valid-lft": 43200 }, reason: ResourceBusy: IP
address:123.123.123.123 could not be updated.)
Mar 12 19:32:05 dhcp-2 kea-dhcp4[203037]: WARN
[kea-dhcp4.lease-cmds-hooks.139670018098944] LEASE_CMDS_UPDATE4_CONFLICT
lease4-update command failed due to conflict (parameters: { "client-id": "",
"expire": 1678689125, "force-create": true, "fqdn-fwd": false, "fqdn-rev":
false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0,
"subnet-id": 274, "valid-lft": 43200 }, reason: ResourceBusy: IP
address:100.72.19.33 could not be updated.)
Mar 12 19:32:34 dhcp-2 kea-dhcp4[203037]: WARN
[kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT
lease4-update command failed due to conflict (parameters: { "client-id": "",
"expire": 1678689154, "force-create": true, "fqdn-fwd": false, "fqdn-rev":
false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0,
"subnet-id": 113, "valid-lft": 43200 }, reason: ResourceBusy: IP
address:123.123.123.123 could not be updated.)
Mar 12 19:32:36 dhcp-2 kea-dhcp4[203037]: ERROR
[kea-dhcp4.ha-hooks.139670104323840] HA_TERMINATED HA service terminated due to
an unrecoverable condition. Check previous error message(s), address the
problem and restart!
Mar 12 22:11:09 dhcp-2 kea-dhcp4[203037]: ERROR
[kea-dhcp4.packets.139670138794688] DHCP4_BUFFER_RECEIVE_FAIL error on attempt
to receive packet: Truncated DHCPv4 packet (len=0) received, at least 236 is
expected.
This is the HA part of the configuration on the standby dhcp server:
"hooks-libraries": [{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so",
"parameters": {}
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [{
"this-server-name": "standby-dhcp",
"mode": "hot-standby",
"heartbeat-delay": 10000,
"max-response-delay": 60000,
"max-ack-delay": 5000,
"max-unacked-clients": 5,
"peers": [{
"name": "primary-dhcp",
"url": "http://dhcp-1:8001/";,
"role": "primary",
"auto-failover": true
}, {
"name": "standby-dhcp",
"url": "http://dhcp-2:8001/";,
"role": "standby",
"auto-failover": true
}]
}]
}
}]
The configuration on dhcp-1 is similar except for the "this-server-name"
attribute.
Thanks,
Sebastian
On Sun Jun 11, 2023 at 12:35 PM CEST, Darren Ankney wrote:
> Hi Sebastian,
>
> Please share the HA messages surrounding and including one of these
> terminations. The termination, or a proceeding message should state
> the cause. The 'ResourceBusy' messages are not warnings I am familiar
> with. Could you also send your HA configuration from each server
> (anonymized of course)?
>
> Thank you,
>
> Darren Ankney
>
> On Sat, Jun 10, 2023 at 11:51 AM sebastians <[email protected]> wrote:
> >
> > Hi all,
> >
> > we recently migrated our DHCP setup from dhcpd to Kea. It runs on
> > two servers with hot standby and a memfile backend for the leases. Kea
> > assigns IP addresses for around 7000 pools.
> >
> > Over the past few months the HA connection terminated in random intervals.
> > From looking at the logs on the passive node I can see a lot of
> > 'ResourceBusy: IP address ... could not be updated' warnings prior to
> > the connection terminating. Since multithreading is enabled I suspected
> > this may be due to the threads encountering a resource lock on the memfile.
> > I suppose after the lease update fails a few times, the connection is
> > terminated.
> >
> > Is the 'ResourceBusy' warning the cause for the terminating HA connection
> > and
> > is there any way to fix the underlying issue? Any ideas on the issue are
> > greatly
> > appraciated.
> >
> > Thanks,
> >
> > Sebastian
> > --
> > ISC funds the development of this software with paid support subscriptions.
> > Contact us at https://www.isc.org/contact/ for more information.
> >
> > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
> >
> > Kea-users mailing list
> > [email protected]
> > https://lists.isc.org/mailman/listinfo/kea-users
> --
> ISC funds the development of this software with paid support subscriptions.
> Contact us at https://www.isc.org/contact/ for more information.
>
> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>
> Kea-users mailing list
> [email protected]
> https://lists.isc.org/mailman/listinfo/kea-users
--
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
Kea-users mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/kea-users
