On Sat, 16 Apr 2016 00:42:33 +0530 Ritesh Raj Sarraf <r...@debian.org> wrote: > On Sat, 2016-04-16 at 00:35 +0530, Ritesh Raj Sarraf wrote: > > Hello Mike, > > > > > All I can say is that I am able to see the behavior that you've mentioned (So > > marking the bug as confirmed). THis is seen on my LIO iSCSI Setup running on 2 > > Guest VMs. > > > > In doing that, I was surprised that those dev/mapper/ entries are now > > symlinks. > > My storage knowledge has rusted for some time now, but from what I recollect, > > it > > used to by direct block node names, not symlinks. > > Maybe this could be of some help to you in debugging this further. > > Apr 16 00:34:11 debian-sanboot systemd[1]: Started Device-Mapper Multipath > Device Controller. > Apr 16 00:34:11 debian-sanboot systemd-udevd[949]: conflicting device node > '/dev/mapper/sanroot' found, link to '/dev/dm-0' will not be created > > > So I think my understanding was correct about Device Mapper's behavior. Now I'm > not sure what 'systemd-udevd[949]' is reflecting here as. I doubt if that is the > ID for the rules processing because then I'd have seen the respective rule. > > I hope this is of some use to you. My Debian time for today is running out :-) >
Hi Ritesh, I am working with Mike on this problem. We have seen this issue using multiple transports (FCoE, FC, and iSCSI), although most of our testing has been using QLogic FC cards with 3Par FC storage. I don't think this is a transport problem, but is instead, some sort of udev/multipath interaction problem. Here is a test I ran with multipath-tools_0.5.0-7 and multipath-tools_0.5.0+git0.770e6d0d-1 on stretch using the same hardware and configuration that Mike has been using: Here is the storage: 1:0:0:1] disk 3PARdata VV 3212 /dev/sdb [1:0:0:2] disk 3PARdata VV 3212 /dev/sdc [1:0:0:254] enclosu 3PARdata SES 3212 - [1:0:1:1] disk 3PARdata VV 3212 /dev/sdd [1:0:1:2] disk 3PARdata VV 3212 /dev/sde [1:0:1:254] enclosu 3PARdata SES 3212 - And udev monitor output (udevadm monitor -u -k): multipath-tools_0.5.0-7 # udevadm monitor -k -u monitor will print the received events for: UDEV - the event which udev sends out after rule processing KERNEL - the kernel uevent KERNEL[90540.185998] change /devices/virtual/block/dm-0 (block) KERNEL[90540.187857] change /devices/virtual/block/dm-1 (block) UDEV [90540.265654] change /devices/virtual/block/dm-1 (block) UDEV [90540.266138] change /devices/virtual/block/dm-0 (block) # ls -l /dev/mapper total 0 crw------- 1 root root 10, 236 Apr 14 14:31 control lrwxrwxrwx 1 root root 7 Apr 15 15:40 mpatha -> ../dm-0 lrwxrwxrwx 1 root root 7 Apr 15 15:40 mpathb -> ../dm-1 multipath-tools_0.5.0+git0.770e6d0d-1 # udevadm monitor -k -u monitor will print the received events for: UDEV - the event which udev sends out after rule processing KERNEL - the kernel uevent KERNEL[91354.622867] change /devices/virtual/block/dm-0 (block) KERNEL[91354.625218] change /devices/virtual/block/dm-1 (block) UDEV [91354.699455] change /devices/virtual/block/dm-0 (block) UDEV [91354.701663] change /devices/virtual/block/dm-1 (block) So the udev events look identical. We think that perhaps we have a race and the new version of multipath is not waiting (or waiting long enough) for udev to crate the device files and just goes and creates the required files. I am going to try and bisect the changes in the upstream multipath code between 0.5.0 and 770e6d0d to see where this breaks. This does not seem to that serious a problem on its own, since the block device that is created is identical to what the symlink would point to. But we are seeing other issue. But we are seeing other issues like device files reappearing after the running multipath -f on it, removing the LUN, and then running multipath -r which we think is related. -- Andrew Patterson Hewlett-Packard Enterprise
