Re: ZFS secondarycache on SSD problem on r255173

2014-06-06 Thread Matthew D. Fuller
On Fri, Apr 25, 2014 at 08:26:52PM -0500 I heard the voice of Matthew D. Fuller, and lo! it spake thus: > On Mon, Mar 03, 2014 at 11:17:05AM +0200 I heard the voice of > Andriy Gapon, and lo! it spake thus: > > > > I noticed that on some of our systems we were getting a clearly > > abnormal number

Re: ZFS secondarycache on SSD problem on r255173

2014-04-25 Thread Matthew D. Fuller
On Mon, Mar 03, 2014 at 11:17:05AM +0200 I heard the voice of Andriy Gapon, and lo! it spake thus: > > I noticed that on some of our systems we were getting a clearly > abnormal number of l2arc checksum errors accounted in l2_cksum_bad. > [...] > I propose the following patch which has been tested

Re: ZFS secondarycache on SSD problem on r255173

2014-03-11 Thread Matthew D. Fuller
On Mon, Mar 03, 2014 at 11:17:05AM +0200 I heard the voice of Andriy Gapon, and lo! it spake thus: > > I noticed that on some of our systems we were getting a clearly > abnormal number of l2arc checksum errors accounted in l2_cksum_bad. > The hardware appeared to be in good health. FWIW, I have a

Re: ZFS secondarycache on SSD problem on r255173

2014-03-03 Thread Steven Hartland
- Original Message - From: "Andriy Gapon" on 18/10/2013 17:57 Steven Hartland said the following: I think we we may well need the following patch to set the minblock size based on the vdev ashift and not SPA_MINBLOCKSIZE. svn diff -x -p sys/cddl/contrib/opensolaris/uts/common/fs/zfs

Re: ZFS secondarycache on SSD problem on r255173

2014-03-03 Thread Andriy Gapon
on 18/10/2013 17:57 Steven Hartland said the following: > I think we we may well need the following patch to set the minblock > size based on the vdev ashift and not SPA_MINBLOCKSIZE. > > svn diff -x -p sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > Index: sys/cddl/contrib/opensolaris/uts/

Re: ZFS secondarycache on SSD problem on r255173

2013-10-22 Thread Vitalij Satanivskij
Ок, just up to now no error on l2arc L2 ARC Summary: (HEALTHY) Passed Headroom:1.99m Tried Lock Failures:144.53m IO In Progress: 130.15k Low Memory Aborts: 7 Free on W

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Vitalij Satanivskij
Steven Hartland wrote: SH> So previously you only started seeing l2 errors after there was SH> a significant amount of data in l2arc? Thats interesting in itself SH> if thats the case. Yes someting arround 200+gb SH> I wonder if its the type of data, or something similar. Do you SH> run compres

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Steven Hartland
So previously you only started seeing l2 errors after there was a significant amount of data in l2arc? Thats interesting in itself if thats the case. I wonder if its the type of data, or something similar. Do you run compression on any of your volumes? zfs get compression Regards Steve ---

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Vitalij Satanivskij
Just now I cannot say, as to triger problem we need at last 200+gb size on l2arc wich usually grow in one production day. But for some reason today in the morning server was rebooted so cache was flushed and now only 100Gb. Need to wait some more time. At last for now none error on l2. S

Re: ZFS secondarycache on SSD problem on r255173

2013-10-21 Thread Steven Hartland
Hows things looking Vitalij? - Original Message - From: "Vitalij Satanivskij" Ok. Just right now system rebooted with you patch. Trim enabled again. WIll wait some time untile size of used cache grow's. Steven Hartland wrote: SH> Looking at the l2arc compression code I believe t

Re: ZFS secondarycache on SSD problem on r255173

2013-10-19 Thread Vitalij Satanivskij
; - Original Message - SH> From: "Vitalij Satanivskij" SH> To: "Steven Hartland" SH> Cc: "Vitalij Satanivskij" ; "Dmitriy Makarov" ; "Justin T. Gibbs" ; "Borja SH> Marcos" ; SH> Sent: Friday, October 18, 2013 3:45

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Steven Hartland
ot;Steven Hartland" Cc: "Vitalij Satanivskij" ; "Dmitriy Makarov" ; "Justin T. Gibbs" ; "Borja Marcos" ; Sent: Friday, October 18, 2013 3:45 PM Subject: Re: ZFS secondarycache on SSD problem on r255173 Just right now stats not to actual because of s

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Vitalij Satanivskij
SH> SH> - Original Message - SH> From: "Vitalij Satanivskij" SH> To: "Steven Hartland" SH> Cc: ; "Justin T. Gibbs" ; ; "Borja Marcos" ; SH> "Dmitriy Makarov" SH> Sent: Friday, October 18, 2013 9:01 AM SH> Subj

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Steven Hartland
Steven Hartland" Cc: ; "Justin T. Gibbs" ; ; "Borja Marcos" ; "Dmitriy Makarov" Sent: Friday, October 18, 2013 9:01 AM Subject: Re: ZFS secondarycache on SSD problem on r255173 Hello. Yesterday system was rebooted with vfs.zfs.trim.enabled=0 System version 10.

Re: ZFS secondarycache on SSD problem on r255173

2013-10-18 Thread Vitalij Satanivskij
Hello. Yesterday system was rebooted with vfs.zfs.trim.enabled=0 System version 10.0-BETA1 FreeBSD 10.0-BETA1 #6 r256669, without any changes in code Uptime 10:51 up 16:41 sysctl vfs.zfs.trim.enabled vfs.zfs.trim.enabled: 0 Around 2 hours ago errors counter's kstat.zfs.misc.arcstats.l2_ck

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Steven Hartland
Correct. - Original Message - From: "Vitalij Satanivskij" Just to be sure I understand you clearly, I need to test next configuration: 1) System with ashift patch eg. just latest stable/10 revision 2) vfs.zfs.trim.enabled=0 in /boot/loader.conf So realy only diferens in default

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Vitalij Satanivskij
: "Vitalij Satanivskij" SH> To: "Steven Hartland" SH> Cc: "Justin T. Gibbs" ; "Vitalij Satanivskij" ; ; "Borja Marcos" SH> ; "Dmitriy Makarov" SH> Sent: Thursday, October 17, 2013 7:12 AM SH> Subject: Re: ZFS secondar

Re: ZFS secondarycache on SSD problem on r255173

2013-10-17 Thread Steven Hartland
Cc: "Justin T. Gibbs" ; "Vitalij Satanivskij" ; ; "Borja Marcos" ; "Dmitriy Makarov" Sent: Thursday, October 17, 2013 7:12 AM Subject: Re: ZFS secondarycache on SSD problem on r255173 Hello. SSD is Intel SSD 530 series (INTEL SSDSC2BW180A4 DC12

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Hello. SSD is Intel SSD 530 series (INTEL SSDSC2BW180A4 DC12) Controler is onboard intel sata controler, motherboard is Supermicro X9SRL-F so it's Intel C602 chipset All cache ssd connected to sata 2 ports. System has LSI MPS controler (SAS2308) with firmware version - 16.00.00.00, but only h

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Hello. Problem description is in - http://lists.freebsd.org/pipermail/freebsd-current/2013-October/045088.html As we find later first begin's problem with errors counter in arcstats than size of l2 grows abnormal. After patch rollback everything is ok. Justin T. Gibbs wrote: JTG> You'll hav

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
Ohh stupid question what hardware are you running this on, specifically what SSD's and what controller and if relavent what controller Firmware version? I wonder if you might have bad HW / FW, such as older LSI mps Firmware, which is know to causing corruption with some delete methods. Without t

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
- Original Message - From: "Justin T. Gibbs" I took a quick look at arc.c and see that the trim_map_free() calls don't take into account ashift. I don't know if that has anything to do with your problem though. I would expect this would just make the trim less efficient, but I need

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Justin T. Gibbs
I took a quick look at arc.c and see that the trim_map_free() calls don't take into account ashift. I don't know if that has anything to do with your problem though. I would expect this would just make the trim less efficient, but I need to dig further. -- Justin On Oct 16, 2013, at 4:42 PM,

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Justin T. Gibbs
You'll have to be more specific. I don't have that email or know what list on which to search. Thanks, Justin On Oct 16, 2013, at 2:01 AM, Vitalij Satanivskij wrote: > Hello. > > Patch brocke cache functionality. > > Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300 > > With s

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Steven Hartland wrote: SH> I'm not clear what you rolled back there as r255173 has ntothing to do SH> with this. Could you clarify r255173 with you patch from email dated Tue, 17 Sep 2013 23:53:12 +0100 with subject Re: ZFS secondarycache on SSD problem on r255173 Errors wich we

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
255753 2. Set vfs.zfs.max_auto_ashift=9 Regards Steve - Original Message - From: "Vitalij Satanivskij" To: "Steven Hartland" Cc: "Vitalij Satanivskij" ; "Dmitriy Makarov" ; "Justin T. Gibbs" ; "Borja Marcos" ; Sent: Wednesday, October 1

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
n Hartland" ; "Justin T. Gibbs" ; "Borja Marcos" ; SH> SH> Sent: Wednesday, October 16, 2013 9:01 AM SH> Subject: Re: ZFS secondarycache on SSD problem on r255173 SH> SH> SH> > Hello. SH> > SH> > Patch brocke cache functionality. SH>

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Steven Hartland
: "Vitalij Satanivskij" To: "Dmitriy Makarov" Cc: "Steven Hartland" ; "Justin T. Gibbs" ; "Borja Marcos" ; Sent: Wednesday, October 16, 2013 9:01 AM Subject: Re: ZFS secondarycache on SSD problem on r255173 Hello. Patch brocke cache functionality

Re: ZFS secondarycache on SSD problem on r255173

2013-10-16 Thread Vitalij Satanivskij
Hello. Patch brocke cache functionality. Look at's Dmitriy's mail from Mon, 07 Oct 2013 21:09:06 +0300 With subject ZFS L2ARC - incorrect size and abnormal system load on r255173 As patch alredy in head and BETA it's not good. Yesterday we update one machine up to beta1 and forgot about patch

Re: ZFS secondarycache on SSD problem on r255173

2013-09-20 Thread Justin T. Gibbs
On Sep 17, 2013, at 4:53 PM, Steven Hartland wrote: > - Original Message - From: "Justin T. Gibbs" > > >> Sorry for being slow to chime in on this thread. I live in Boulder, CO and >> we've had a bit of rain. :-) > > Hope all is well your side, everyone safe and sound if may be litt

Re: ZFS secondarycache on SSD problem on r255173

2013-09-17 Thread Steven Hartland
- Original Message - From: "Justin T. Gibbs" Sorry for being slow to chime in on this thread. I live in Boulder, CO and we've had a bit of rain. :-) Hope all is well your side, everyone safe and sound if may be little wetter than usual. As Steven pointed out, the warning is ben

Re: ZFS secondarycache on SSD problem on r255173

2013-09-16 Thread Justin T. Gibbs
Sorry for being slow to chime in on this thread. I live in Boulder, CO and we've had a bit of rain. :-) As Steven pointed out, the warning is benign, but does show that the code I committed to -current is not optimizing the allocation size for L2ARC devices. The fix for this is to find the be

Re: ZFS secondarycache on SSD problem on r255173

2013-09-16 Thread Steven Hartland
"Dmitryy Makarov" ; ; "Justin T. Gibbs" Sent: Monday, September 16, 2013 12:06 PM Subject: Re: ZFS secondarycache on SSD problem on r255173 On Sep 13, 2013, at 2:18 PM, Steven Hartland wrote: This is a recent bit of code by Justin cc'ed, so he's likely the best pe

Re: ZFS secondarycache on SSD problem on r255173

2013-09-16 Thread Borja Marcos
On Sep 13, 2013, at 2:18 PM, Steven Hartland wrote: > This is a recent bit of code by Justin cc'ed, so he's likely the best person > to > investigate this one. Hmm. There is still a lot of confusion surrounding all this, and it's a time bomb waiting to explode. A friend had serious problems o

Re: ZFS secondarycache on SSD problem on r255173

2013-09-13 Thread Steven Hartland
This is a recent bit of code by Justin cc'ed, so he's likely the best person to investigate this one. Regards Steve - Original Message - From: "Dmitryy Makarov" To: Sent: Friday, September 13, 2013 12:16 PM Subject: ZFS secondarycache on SSD problem on r255173 Hello, FreeBSD C