It may be that your logs are rotating too quickly after seeing your logging configuration. There may be a large amount of logs in a short period of time during trouble events. depending how many clients you have. According to https://kea.readthedocs.io/en/kea-2.3.2/arm/logging.html maxsize is in bytes, so you've got it set to 200KB logfile size. maxver defines how many rotated files to keep. So you could keep a maximum of 1.5MB of logs (unless my math is off). We get these messages every 10 seconds:
COMMAND_RECEIVED Received command 'ha-heartbeat' during normal operation. I think you won't have a very long record of what is happening with the server with these settings. 06:32:57.393 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is UNDEFINED It looks like adc2 lost connection to adc1 (you said adc2 was the primary). I guess that may have been the point that your onsite administrator unplugged adc1. Is there anything that is not labeled kea-dhcp4.ha-hooks in the logfile earlier than this? On Wed, Nov 23, 2022 at 6:52 AM Stefan G. Weichinger <[email protected]> wrote: > > Am 23.11.22 um 12:43 schrieb Darren Ankney: > > > If clients were attempting to get addresses at what should have been > > an elevated pace, there should have been elevated logs in the minutes > > leading up to the unplug. There are many nuanced logging options with > > Kea. Do you normally see detailed logs of clients attempting to > > retrieve addresses (successful or unsuccessful)? If you had no logs > > at all, then the logging may be broken, not configured correctly in > > the first place, client messages may not have been getting to the > > server, or Kea may not have been running. > > I am quite sure it was running. Browsed the logs up to that plugging > event, communication between the nodes was up. > > I grepped this out of the logs, maybe something is visible here: > > 06:32:57.393 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is > UNDEFINED > 06:39:50.404 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from HOT-STANDBY to WAITING state, partner state is PARTNER-DOWN > 06:39:50.404 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN > 06:39:50.441 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from SYNCING to READY state, partner state is PARTNER-DOWN > 06:39:55.403 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from PARTNER-DOWN to HOT-STANDBY state, partner state is READY > 06:40:54.795 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from READY to PARTNER-DOWN state, partner state is UNDEFINED > 06:41:06.394 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is > UNDEFINED > 07:12:28.853 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from PARTNER-DOWN to WAITING state, partner state is > PARTNER-DOWN > 07:12:28.853 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN > 07:12:28.879 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from SYNCING to READY state, partner state is PARTNER-DOWN > 07:12:35.392 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from PARTNER-DOWN to WAITING state, partner state is READY > 07:12:35.392 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from WAITING to SYNCING state, partner state is READY > 07:12:35.420 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from READY to HOT-STANDBY state, partner state is READY > 07:12:35.420 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from SYNCING to READY state, partner state is READY > 07:12:39.060 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from READY to HOT-STANDBY state, partner state is HOT-STANDBY > 07:23:04.570 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is > UNDEFINED > 07:24:13.921 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server > transitions from WAITING to PARTNER-DOWN state, partner state is UNDEFINED > > > >> > >> My assumption: > >> > >> the samba-installation might have removed or replaced some package kea > >> depends on. I don't know what ... > > > > Most likely, Kea would have failed to run at all if some libraries > > were unavailable or were incorrect versions (assuming Linux here). > > Yes, as mentioned: Debian 11 Bullseye. > > Good to know. That was just one thought, as always I try to think "what > has changed?" to analyse ... > > >> so maybe kea on adc2 was already not working OK anymore yesterday > >> evening and in the night ... but that wasn't noticed because the leases > >> were still active (?) > > > > That is possible. Depending on your lease length, you wouldn't > > necessarily notice anything until someone complained about being > > unable to obtain an IP address. > > yes > > > Before you rebooted adc2, did you check if Kea was running? > > it was running > > > Is this > > DHCPv4, DHCPv6 or both? What OS is adc2? Which HA mode did you > > choose? > > Debian 11.5 > isc-kea-2.3.2 > DHCPv4 only > hot-standby (with adc2 as primary) > > > If you would share your logging configuration, that would help > > too. > > Sure: > > currently the loglevel is pretty low, I assume. I had "severity" : > "DEBUG" while the setup phase. > > -> > > "loggers": [ > { > "name": "kea-dhcp4", > "output_options": [ > { > "output": "/var/log/kea/kea-debug.log", > "maxver": 8, > "maxsize": 204800, > "flush": true, > "pattern": "%d{%j %H:%M:%S.%q} %c %m\n" > } > ], > # "severity": "DEBUG", > "severity": "INFO", > "debuglevel": 99 > } > ] > > thanks a lot > > -- > ISC funds the development of this software with paid support subscriptions. > Contact us at https://www.isc.org/contact/ for more information. > > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. > > Kea-users mailing list > [email protected] > https://lists.isc.org/mailman/listinfo/kea-users -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list [email protected] https://lists.isc.org/mailman/listinfo/kea-users
