** Description changed: This is a tracker for upstream bugs https://github.com/containerd/containerd/issues/13609 and https://github.com/containerd/containerd/issues/13355 - Description (from upstream bug): + [ Impact ] - When NRI RunPodSandbox fails after StartSandbox has already created a - container record in bolt DB, the deferred cleanup path does not delete - the container. This can be triggered by NRI plugins rejecting the - sandbox after it has been started, as well as CNI setup timeouts, ttrpc - deadline errors, or other failures that occur after StartSandbox. If - kubelet retries the same pod and succeeds, multiple containers with the - same sandbox name but different IDs coexist in bolt DB. On containerd - restart, recover() calls sandboxNameIndex.Reserve() for each container - and hits a name conflict, causing a fatal exit that makes the node - completely unusable. + In a kubernetes cluster, when NRI RunPodSandbox fails after StartSandbox + has already created a container record in bolt DB, the deferred cleanup + path does not delete the container. If kubelet retries the same pod and + succeeds, multiple containers with the same sandbox name but different + IDs coexist in bolt DB. On containerd restart, recover() calls + sandboxNameIndex.Reserve() for each container and hits a name conflict, + causing a fatal exit that makes the node completely unusable. + + [ Test plan ] + + Since there are no minimal reproducers available for this bug, and + considering it requires complex k8s setups, this will need to be tested + by affected users in order to be verified. + + [ Where problems could occur ] + + The new ShutdownSandbox rollback could change the pod creation error + handling codepath, leading to hung or delayed pod creation operations in + the cases where an error occurs. + + The new ShutdownSandbox rollback adds a cleanup step to the pod creation + error codepath. This could lead to delays to RunPodSandbox operations + (or make them hang) in cases where an error occurs. This does not affect + successful pod creation operations. + + [ Other info ]
** Description changed: This is a tracker for upstream bugs https://github.com/containerd/containerd/issues/13609 and https://github.com/containerd/containerd/issues/13355 [ Impact ] In a kubernetes cluster, when NRI RunPodSandbox fails after StartSandbox has already created a container record in bolt DB, the deferred cleanup path does not delete the container. If kubelet retries the same pod and succeeds, multiple containers with the same sandbox name but different IDs coexist in bolt DB. On containerd restart, recover() calls sandboxNameIndex.Reserve() for each container and hits a name conflict, causing a fatal exit that makes the node completely unusable. [ Test plan ] Since there are no minimal reproducers available for this bug, and considering it requires complex k8s setups, this will need to be tested by affected users in order to be verified. [ Where problems could occur ] The new ShutdownSandbox rollback could change the pod creation error handling codepath, leading to hung or delayed pod creation operations in the cases where an error occurs. The new ShutdownSandbox rollback adds a cleanup step to the pod creation error codepath. This could lead to delays to RunPodSandbox operations (or make them hang) in cases where an error occurs. This does not affect successful pod creation operations. - - [ Other info ] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170996 Title: containerd fatal on restart To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd-app/+bug/2170996/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
