Hi Tim, I don't think patching kubernetes-master will resolve this issue. Mainly because the issue happens on kubernetes-workers.
We were seeing PVCs failing with: "No VM found". When we look into the logs, actually kubernetes was trying to learn about its kubernetes- workers through SystemUUID. Essentially: https://github.com/kubernetes/kubernetes/blob/103e926604de6f79161b78af3e792d0ed282bc06/staging/src/k8s.io/legacy-cloud-providers/vsphere/vsphere_util.go#L656 Recovers UUID from SystemUUID, which according to https://github.com/kubernetes/kubernetes/issues/58927 Comes from product_serial However, because of this dmidecode issue, the first half of the bytes are inversed (big endian vs. small endian). What we saw on kubernetes-controller-manager log then was k8s grabbing a half-wrong UUID and trying to fetch it using FindVMUUID method. Of course that broke, since product_uuid held the right values. To fix that, we have two ways: 1) work-around: run VMs with compatibility set for ESXi 5.5 and later; that forces vsphere to setup those VMs with older HW version (10) and there is no issue with dmidecode 2) Find out why dmidecode diverges when going to version 13 and fix it -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to dmidecode in Ubuntu. https://bugs.launchpad.net/bugs/1851676 Title: dmidecode decodes /sys/class/dmi/id/product_serial incorrectly Status in dmidecode package in Ubuntu: New Bug description: Hi, On VMWare 6.5 and higher (HW version 13 and higher) when decoding /sys/class/dmi/id/product_serial dmidecode produces an output whereby the first 16 characters have been reversed in comparison to /sys/class/dmi/id/product_uuid. This manifests itself by breaking provisioning in Kubernetes and could cause other issues down the line. Other OS such as redhat have already patched this in their releases. A demonstration of the issue can be found in this pastebin: https://pastebin.canonical.com/p/qRfrpHXwGs/ VMWare Issue: https://kb.vmware.com/s/article/53609 Cheers! To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/dmidecode/+bug/1851676/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
