[Kernel-packages] [Bug 1470250] Re: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups
So I'm the OP on this bug report from what seems like ages ago... While a lot of hard work as been done by some of the members posting here to try and replicate it, it's been very difficult to track the specific event that has been causing this.While we "fixed" our problem a long time ago by moving all the VM's effected to a server and disabling Hyper-V nightly snapshots, instead opting to shut down all VM's once a month on a weekend and shapshot the whole system and bring it back online, it hasn't been optimal. Lately, we've been upgrading the 14.04LTS systems one at a time to 16.04 with the latest kernel and dropping them back into active nightly backup systems and so far, have not seen a read-only file system failure with systems running the latest updates. Now these systems are not write heavy either, which tends to trigger this bug more often. At least we've seen some improvement in our environment just by keeping up to date with the latest distribution updates. Outside of this, microsoft has released server 2016 during our lifetime, and while we are not running, it would be interesting to see if that Hyper-V server OS is any better/worse then 2012R2. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1470250 Title: [Hyper-V] Ubuntu 14.04.2 LTS Generation 2 SCSI Errors on VSS Based Backups Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Status in linux source package in Zesty: In Progress Bug description: Customers have reported running various versions of Ubuntu 14.04.2 LTS on Generation 2 Hyper-V Hosts.On a random Basis, the file system will be mounted Read-Only due to a "disk error" (which really isn't the case here).As a result, they must reboot the Ubuntu guest to get the file system to mount RW again. The Error seen are the following: Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968142] storvsc: Sense Key : Unit Attention [current] Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968145] storvsc: Add. Sense: Changed operating definition Apr 30 00:02:01 balticnetworkstraining kernel: [640153.968161] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584164] hv_storvsc vmbus_0_4: cmd 0x2a scsi status 0x2 srb status 0x82 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584178] hv_storvsc vmbus_0_4: stor pkt 88006eb6c700 autosense data valid - len 18 Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584180] storvsc: Sense Key : Unit Attention [current] Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584183] storvsc: Add. Sense: Changed operating definition Apr 30 01:23:26 balticnetworkstraining kernel: [645039.584198] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. This relates to the VSS "Windows Server Backup" process that kicks off at midnight on the host and finishes an hour and half later. Yes, we do have hv_vss_daemon and hv_kvp_daemon running for the correct kernel version we have. We're currently running kernel version 3.13.0-49-generic #83 on one system and 3.16.0-34-generic #37 on the other. -- We see the same errors on both. As a result, we've been hesitant to drop any more ubuntu guests on our 2012R2 hyper-v system because of this. We can stop the backup process and all is good, but we need nightly backups to image all of our VM's. All the windows guests have no issues of course. We also have some CentOS based guests running without issues from what we've seen. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1470250/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1445195] Re: [Hyper-V] Kernel patches for storvsc
I downloaded the 3.16-39 kernels and files and applied them to a test system we have running on Hyper-V.I'm not able to get the -39 kernel to fully boot. The startup hangs on the following line: EXT4-fs (sda2) re-mounted. Opts: errors=remount-ro If I restart the system with kernel -38, I can boot, but of course without the patches here.I'm running 14.04 LTS with the 3.16 kernel, but I could try the 3.13 kernel or 3.19. Another problem with the install is that linux-tools required a different REV of binutils, but from what I could tell, it wanted a rev lower then what I had, I ignored it- -- but that shouldn't be the reason the system won't finish mounting the file system and booting. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1445195 Title: [Hyper-V] Kernel patches for storvsc Status in linux package in Ubuntu: Triaged Status in linux source package in Precise: Triaged Status in linux source package in Trusty: Triaged Status in linux source package in Utopic: Triaged Status in linux source package in Vivid: Triaged Status in linux source package in Wily: Triaged Bug description: Storage driver performance updates for vivid K. Y. Srinivasan (7): scsi: storvsc: Increase the ring buffer size scsi: storvsc: Size the queue depth based on the ringbuffer size scsi: storvsc: Always send on the selected outgoing channel scsi: storvsc: Retrieve information about the capability of the target scsi: storvsc: Fix a bug in copy_from_bounce_buffer() scsi: storvsc: Don't assume that the scatterlist is not chained scsi: storvsc: Set the tablesize based on the information given by the host To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1445195/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1445195] Re: [Hyper-V] Kernel patches for storvsc
Thanks! So, 3.19 kernel runs fine. Something is wrong with the 3.16 build.I show the VSS and KVP daemons registered as well. I'm going to let this run a few days and see what happens. I should start to see the errors during VSS backup tonight if they occur. I may apply this kernel to a few other production VM's that on a regular basis go into RO filesystem mode overnight, but I'll run on our test VM for now. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1445195 Title: [Hyper-V] Kernel patches for storvsc Status in linux package in Ubuntu: Triaged Status in linux source package in Precise: Triaged Status in linux source package in Trusty: Triaged Status in linux source package in Utopic: Triaged Status in linux source package in Vivid: Triaged Status in linux source package in Wily: Triaged Bug description: Storage driver performance updates for vivid K. Y. Srinivasan (7): scsi: storvsc: Increase the ring buffer size scsi: storvsc: Size the queue depth based on the ringbuffer size scsi: storvsc: Always send on the selected outgoing channel scsi: storvsc: Retrieve information about the capability of the target scsi: storvsc: Fix a bug in copy_from_bounce_buffer() scsi: storvsc: Don't assume that the scatterlist is not chained scsi: storvsc: Set the tablesize based on the information given by the host To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1445195/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1445195] Re: [Hyper-V] Kernel patches for storvsc
1st Night Backups Ran okay, with no strange SCSI errors. The following was logged into syslog: May 28 00:01:22 test Hyper-V VSS: VSS: op=FREEZE: succeeded May 28 00:01:21 test kernel: [22188.912452] sd 0:0:0:0: [storvsc] Sense Key : Unit Attention [current] May 28 00:01:21 test kernel: [22188.912465] sd 0:0:0:0: [storvsc] Add. Sense: Changed operating definition May 28 00:01:21 test kernel: [22188.912596] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. May 28 00:01:21 test Hyper-V VSS: VSS: op=THAW: succeeded May 28 00:17:01 test CRON[1323]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) May 28 01:17:01 test CRON[1329]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) May 28 01:36:17 test kernel: [27884.812212] sd 0:0:0:0: [storvsc] Sense Key : Unit Attention [current] May 28 01:36:17 test kernel: [27884.812229] sd 0:0:0:0: [storvsc] Add. Sense: Changed operating definition May 28 01:36:17 test kernel: [27884.812361] sd 0:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. So we're still getting the messages about, hey, something changed on the Disk, but I can live with that for the price of a proper VSS snapshot. Again, this is with the 3.19 -39 kernel on 14.04.2 LTS. I'll keep an eye on it. I guess I'll wait for production changes, but it's so tempting :-) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1445195 Title: [Hyper-V] Kernel patches for storvsc Status in linux package in Ubuntu: Triaged Status in linux source package in Precise: Triaged Status in linux source package in Trusty: Triaged Status in linux source package in Utopic: Triaged Status in linux source package in Vivid: Triaged Status in linux source package in Wily: Triaged Bug description: Storage driver performance updates for vivid K. Y. Srinivasan (7): scsi: storvsc: Increase the ring buffer size scsi: storvsc: Size the queue depth based on the ringbuffer size scsi: storvsc: Always send on the selected outgoing channel scsi: storvsc: Retrieve information about the capability of the target scsi: storvsc: Fix a bug in copy_from_bounce_buffer() scsi: storvsc: Don't assume that the scatterlist is not chained scsi: storvsc: Set the tablesize based on the information given by the host To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1445195/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1445195] Re: [Hyper-V] Kernel patches for storvsc
I wanted to give an update into testing the Vivid kernel on 14.04 LTS re: https://bugs.launchpad.net/bugs/1454758 Our non-critical database production server ran for almost two weeks, but this morning finally died when the file system went RO overnight due to VSS backups. This is with the vivid-19 kernel patches posted here. This also confirms @f-bosch findings that he still is having problems with the new kernel.I went back to https://bugs.launchpad.net/bugs/1454758 and it looks like it's still listed a a duplicate of this bug ID. Being that @jsalisbury wants to keep this on topic for performance improvements, I'm not sure where that leaves us who are more concerned about the reliability aspects of these improvements over the performance. Looking for best next steps on this one. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1445195 Title: [Hyper-V] Kernel patches for storvsc Status in linux package in Ubuntu: Triaged Status in linux source package in Precise: Triaged Status in linux source package in Trusty: Triaged Status in linux source package in Utopic: Triaged Status in linux source package in Vivid: Fix Committed Status in linux source package in Wily: Triaged Bug description: Storage driver performance updates for vivid K. Y. Srinivasan (7): scsi: storvsc: Increase the ring buffer size scsi: storvsc: Size the queue depth based on the ringbuffer size scsi: storvsc: Always send on the selected outgoing channel scsi: storvsc: Retrieve information about the capability of the target scsi: storvsc: Fix a bug in copy_from_bounce_buffer() scsi: storvsc: Don't assume that the scatterlist is not chained scsi: storvsc: Set the tablesize based on the information given by the host To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1445195/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
