Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Vincent Lefevre
On 2015-04-27 11:04:51 -0500, David Wright wrote: > Exactly. Saying "But this matters in a POSIX-compliant filesystem" > is very different from "But with the current solution". To me, > a "current solution" is an concrete implementation, not a POSIX > specification; the latter is a target. I reall

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Bob Proulx
Vincent Lefevre wrote: > In short, ext3 is bad for renames, but if Bob Proulx's suggestion > were implemented (without a workaround such that caching the whole > directory after it is opened[*]), it would be much worse as arbitrary > entries would be missed in a readdir sequence. To be clear I did

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Bob Proulx
David Wright wrote: > Quoting Kushal Kumaran: > > Bob Proulx writes: > > > If a directory became full it was easy to extend it > > > by writing the array longer. But if an early entry in the array was > > > deleted the system would zero it out rather than move each and every > > > entry in the fil

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-26 13:26:41 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > On 2015-04-24 22:01:37 -0500, David Wright wrote: > > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > > So, I would say that this is a

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Vincent Lefevre
On 2015-04-26 13:26:41 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-24 22:01:37 -0500, David Wright wrote: > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > So, I would say that this is a bug. POSIX says in > > > > http://pubs.opengroup.org/onl

Re: reading an empty directory after reboot is very slow

2015-04-26 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-24 22:01:37 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > So, I would say that this is a bug. POSIX says in > > > http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html > > > > It may we

Re: reading an empty directory after reboot is very slow

2015-04-26 Thread Vincent Lefevre
On 2015-04-24 22:01:37 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > So, I would say that this is a bug. POSIX says in > > http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html > > It may well be. But I'm just presenting facts about the > filesyste

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-24 19:41:03 -0500, David Wright wrote: > > Well, that's what I thought, because of the cacheing. But Nicolas > > asked me to try using thousands of files, and so here we are, > > ie, your "new test": > > > > ~ $ for j in `seq 1` ; do m

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-24 19:41:03 -0500, David Wright wrote: > Well, that's what I thought, because of the cacheing. But Nicolas > asked me to try using thousands of files, and so here we are, > ie, your "new test": > > ~ $ for j in `seq 1` ; do mkdir /tmp/testdir/file$j ; done > ~ $ ./a.out /tmp/testdir

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-23 11:22:11 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > On 2015-04-23 11:31:27 +0200, Nicolas George wrote: > > > > Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : > > > > > David's test s

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-24 09:35:29 +0200, Vincent Lefevre wrote: > Anyway remember that the main slowness I have is when I retag messages. > I could improve my script to cache the message-id -> filename mapping, > which would make it much faster. But what I like to understand is why > my current script is so m

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-23 11:22:11 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-23 11:31:27 +0200, Nicolas George wrote: > > > Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : > > > > David's test shows that the renamed file is missed. > > > > > > No, i

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-23 10:10:54 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-22 23:28:46 -0500, David Wright wrote: > > > No, I wasn't expecting mutt to use mairix. But I thought you might be > > > using it. Otherwise, why do you index them? > > > > I use mairix

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-22 23:06:47 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > On 2015-04-21 11:05:58 -0500, David Wright wrote: > > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > > On 2015-04-20 13:04:41 -0500, D

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-23 11:31:27 +0200, Nicolas George wrote: > > Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : > > > David's test shows that the renamed file is missed. > > > > No, it shows that the renamed file is NOT missed: he renamed the en

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Nicolas George (geo...@nsup.org): > Le tridi 3 floréal, an CCXXIII, David Wright a écrit : > > OK. Here's a demonstration of a file going AWOL by moving *up* the > > directory listing. Because of read-ahead, readdir still sees the old > > name and the stat() fails. > > What are you trying

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-22 23:28:46 -0500, David Wright wrote: > > No, I wasn't expecting mutt to use mairix. But I thought you might be > > using it. Otherwise, why do you index them? > > I use mairix when I need a "body" search first, otherwise such a > search

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-23 11:31:27 +0200, Nicolas George wrote: > Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : > > David's test shows that the renamed file is missed. > > No, it shows that the renamed file is NOT missed: he renamed the entry for > inode 497003 from file2 into a long name, and

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Nicolas George
Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : > David's test was to show what could happen in practice after some > simple change, such as removing objects from the directory, not to > take care of all possible race conditions Simple changes are the bricks for race conditions. >

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-23 09:13:02 +0200, Nicolas George wrote: > Le tridi 3 floréal, an CCXXIII, David Wright a écrit : > > OK. Here's a demonstration of a file going AWOL by moving *up* the > > directory listing. Because of read-ahead, readdir still sees the old > > name and the stat() fails. > > What are y

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-22 23:06:47 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-21 11:05:58 -0500, David Wright wrote: > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > On 2015-04-20 13:04:41 -0500, David Wright wrote: > > > > > Quoting Vincent Lefevre (vi

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-22 23:30:49 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-21 10:35:12 -0600, Bob Proulx wrote: > > > Vincent Lefevre wrote: > > > > So, there's still something I don't understand: after dropping the > > > > caches, why is Mutt fast to read the

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-22 23:28:46 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-21 12:47:14 -0500, David Wright wrote: > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > This mailbox is constantly open in a Mutt running in screen (in > > > > read-only mode)

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Nicolas George
Le tridi 3 floréal, an CCXXIII, David Wright a écrit : > OK. Here's a demonstration of a file going AWOL by moving *up* the > directory listing. Because of read-ahead, readdir still sees the old > name and the stat() fails. What are you trying to prove with that test? You would get the same failu

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-21 10:35:12 -0600, Bob Proulx wrote: > > Vincent Lefevre wrote: > > > So, there's still something I don't understand: after dropping the > > > caches, why is Mutt fast to read the mailbox (about 1 minute), but > > > not my tool (about 30 mi

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-21 12:47:14 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > This mailbox is constantly open in a Mutt running in screen (in > > > read-only mode). I often read it, and I modify it from time to time, > > >

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-21 11:05:58 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > On 2015-04-20 13:04:41 -0500, David Wright wrote: > > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > > But with the current solution (

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread Vincent Lefevre
On 2015-04-21 12:47:14 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > This mailbox is constantly open in a Mutt running in screen (in > > read-only mode). I often read it, and I modify it from time to time, > > either by adding new messages in the usual way, or by mo

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread Vincent Lefevre
On 2015-04-21 10:35:12 -0600, Bob Proulx wrote: > Vincent Lefevre wrote: > > So, there's still something I don't understand: after dropping the > > caches, why is Mutt fast to read the mailbox (about 1 minute), but > > not my tool (about 30 minutes)? > > Are you using mutt's header_cache feature?

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread Vincent Lefevre
On 2015-04-21 11:05:58 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > On 2015-04-20 13:04:41 -0500, David Wright wrote: > > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > > But with the current solution (no automatic moving of an entry), you > > > > can't miss

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-20 15:59:22 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > Possibly, but individual modifications would take much more time than > > > with Maildir (such modifications, consisting in retagging, occur from

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Bob Proulx
Vincent Lefevre wrote: > So, there's still something I don't understand: after dropping the > caches, why is Mutt fast to read the mailbox (about 1 minute), but > not my tool (about 30 minutes)? Are you using mutt's header_cache feature? It keeps a cache of mail files off to the side. It really

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-20 13:04:41 -0500, David Wright wrote: > > Quoting Vincent Lefevre (vinc...@vinc17.net): > > > But with the current solution (no automatic moving of an entry), you > > > can't miss an entry that hasn't been removed. [...] > > ...so if you h

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Vincent Lefevre
On 2015-04-20 18:03:01 -0600, Bob Proulx wrote: > Vincent Lefevre wrote: > > With an uncompressed mbox file, using the Content-Length, it could be > > faster, but there's still the problem with individual changes. > > Have you run across Jamie Zawinski's rant against Content-Length? > > http://

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Vincent Lefevre
On 2015-04-20 15:59:22 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > Possibly, but individual modifications would take much more time than > > with Maildir (such modifications, consisting in retagging, occur from > > time to time). > > I take it these are real emai

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Vincent Lefevre
On 2015-04-20 13:04:41 -0500, David Wright wrote: > Quoting Vincent Lefevre (vinc...@vinc17.net): > > But with the current solution (no automatic moving of an entry), you > > can't miss an entry that hasn't been removed. > > $ mkdir /tmp/testdir > ~ $ mkdir /tmp/testdir/file1 > ~ $ mkdir /tmp/test

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread Bob Proulx
Vincent Lefevre wrote: > With an uncompressed mbox file, using the Content-Length, it could be > faster, but there's still the problem with individual changes. Have you run across Jamie Zawinski's rant against Content-Length? http://www.jwz.org/doc/content-length.html > > But let's say that al

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-15 13:41:20 -0600, Bob Proulx wrote: > > Vincent Lefevre wrote: > > > I also notice slowness with a large maildir directory: > > > > > > drwx-- 2 vlefevre vlefevre 8409088 2015-03-24 14:04:33 > > > Mail/oldarc/cur/ > > > > > > In thi

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-15 14:13:43 -0500, David Wright wrote: > > Quoting Kushal Kumaran (kus...@locationd.net): > > > Moving entries around breaks ongoing readdir operations. If a readdir > > > has gone past the file being removed, and you moved the last entry

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread Vincent Lefevre
On 2015-04-15 14:13:43 -0500, David Wright wrote: > Quoting Kushal Kumaran (kus...@locationd.net): > > Moving entries around breaks ongoing readdir operations. If a readdir > > has gone past the file being removed, and you moved the last entry > > there, the entry being moved would be missed, desp

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread Vincent Lefevre
On 2015-04-15 13:41:20 -0600, Bob Proulx wrote: > Vincent Lefevre wrote: > > I also notice slowness with a large maildir directory: > > > > drwx-- 2 vlefevre vlefevre 8409088 2015-03-24 14:04:33 Mail/oldarc/cur/ > > > > In this one, the files are real (145400 files), but I have a Perl > > scr

Re: reading an empty directory after reboot is very slow

2015-04-16 Thread Vincent Lefevre
On 2015-04-15 14:19:40 -0500, David Wright wrote: > Good to see people testing their tools. Actually it is not a test of a tool, but a test to find for various double-precision functions, the arguments that are the hardest to round, in order to solve the Table Maker's Dilemma. Such values can also

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Henrique de Moraes Holschuh wrote: > Bob Proulx wrote: > > Henrique de Moraes Holschuh wrote: > > > 2. unmount filesystem and run e2fsck -D on it every so often. > > > > Every so often? That phrase roused my interest. Wouldn't exactly > > once be enough? > > Yes, as long as nobody ever creates

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Henrique de Moraes Holschuh
On Wed, Apr 15, 2015, at 16:57, Bob Proulx wrote: > Henrique de Moraes Holschuh wrote: > > 2. unmount filesystem and run e2fsck -D on it every so often. > > Every so often? That phrase roused my interest. Wouldn't exactly > once be enough? Yes, as long as nobody ever creates a massive number of

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Henrique de Moraes Holschuh wrote: > 2. unmount filesystem and run e2fsck -D on it every so often. Every so often? That phrase roused my interest. Wouldn't exactly once be enough? I think there are two issues under discussion. One is if dir_index has been applied. For that once would be enoug

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Vincent Lefevre wrote: > Filesystem created: Mon Jan 4 16:26:16 2010 > > My machine is old, but I've never changed anything concerning the > file system. 2010 isn't that old. Just a baby! :-) After a quick look on my network I located these. desolation: Filesystem created: Tue

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Cam Hutchison wrote: > I don't think dir_index has anything to do with it. An index speeds up > lookups. You are not doing lookups; you are traversing the entire data > structure. I think you might be right about Vincent's problem. It seems there is a large amount of uncached data that needs to b

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > On 2015-04-13 15:50:40 -0500, David Wright wrote: > > That's staggering. My /var/lib/dpkg/info has ~8900 files and occupies > > 462848 bytes. So that would be over ½million files in your case. > > Does eftests stand for "excessive files tests"! > > I

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread David Wright
Quoting Kushal Kumaran (kus...@locationd.net): > Bob Proulx writes: > > Petter Adsen wrote: > >> Can someone please enlighten me as to why the entry for this directory > >> is so large, even though it is empty? Since it's apparently obvious to > >> everyone else, I would very much like to know :)

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Vincent Lefevre
On 2015-04-15 09:32:24 -0300, Henrique de Moraes Holschuh wrote: > On Tue, Apr 14, 2015, at 23:18, Vincent Lefevre wrote: > > Can't disk caching be as fast as tmpfs (or almost)? > > Never subestimate the needs of a persistent filesystem to care for > metadata and data safety, unless it has an "eat

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 23:18, Vincent Lefevre wrote: > On 2015-04-14 13:26:16 -0300, Henrique de Moraes Holschuh wrote: > > Yeah, that's a bad habit to have as it slows down way too many > > utilities (lots of stuff benefit for extremely lightweight > > ultra-fast tmpfs in /tmp and $TMPDIR, from

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Vincent Lefevre
On 2015-04-15 12:29:49 +0530, Kushal Kumaran wrote: > Moving entries around breaks ongoing readdir operations. If a readdir > has gone past the file being removed, and you moved the last entry > there, the entry being moved would be missed, despite *it* not being the > entry added or removed. Ind

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Kushal Kumaran
Bob Proulx writes: > Petter Adsen wrote: >> Can someone please enlighten me as to why the entry for this directory >> is so large, even though it is empty? Since it's apparently obvious to >> everyone else, I would very much like to know :) > > > > If a directory became full it was easy to exte

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 23:01:19 -, Cam Hutchison wrote: > I don't think dir_index has anything to do with it. An index speeds up > lookups. You are not doing lookups; you are traversing the entire data > structure. A B-tree data structure can take longer to traverse than a > contiguous array data struct

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread The Wanderer
On 04/14/2015 at 12:55 PM, Bob Proulx wrote: > Petter Adsen wrote: > >> Can someone please enlighten me as to why the entry for this >> directory is so large, even though it is empty? Since it's >> apparently obvious to everyone else, I would very much like to know >> :) > > In the old days dire

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 13:26:16 -0300, Henrique de Moraes Holschuh wrote: > Yeah, that's a bad habit to have as it slows down way too many > utilities (lots of stuff benefit for extremely lightweight > ultra-fast tmpfs in /tmp and $TMPDIR, from "sort" to gcc without > -pipe), but it is indeed widespread. C

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 10:55:12 -0600, Bob Proulx wrote: > Other file systems such as xfs designed for large files and large > numbers of files DO shrink when files are removed. That is one of the > reasons why xfs is recommended for industrial strength use. It was > designed to handle those kinds of work

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Cam Hutchison
Vincent Lefevre writes: >On 2015-04-13 16:28:27 -0600, Bob Proulx wrote: >> Without dir_index an ext filesystem with large directories is slow due >> to the linear nature of directories. But with dir_index it should be >> using a B-tree data structure and should be much faster. >So, why is it

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 14:52, Doug wrote: > >>> Removing files from the directory does not change directory's inode > >>> size. If using ext4, at least. > What is the best way of solving this? Since the directory is empty, can you > just do an rmdir dirname ? Now if there are one or two files i

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Doug
On 04/14/2015 05:15 AM, Reco wrote: Hi. On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: /snip/ 2015-04-13 14:39 GMT+02:00 Vincent Lefevre : The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and 40 seconds). It has been reproducib

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 11:04:06 -0300, Henrique de Moraes Holschuh wrote: > On Tue, Apr 14, 2015, at 10:06, Vincent Lefevre wrote: > > > Without dir_index an ext filesystem with large directories is slow due > > > to the linear nature of directories. But with dir_index it should be > > > using a B-tree dat

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Bob Proulx
Petter Adsen wrote: > Can someone please enlighten me as to why the entry for this directory > is so large, even though it is empty? Since it's apparently obvious to > everyone else, I would very much like to know :) In the old days directories were simply an array of fixed sized integers and fixe

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 13:11, Bob Proulx wrote: > Oh! I was unaware of the e2fsck -D option. Good deal! I didn't recall it existed either, or I'd have mentioned it much earlier in the thread... but I noticed it in the manpage while doing some research due to my participation on this thread.

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Bob Proulx
Henrique de Moraes Holschuh wrote: > Bob Proulx wrote: > > tune2fs -O dir_index /dev/sda5 > > But existing directories are not converted. Only new directories are > > Do a forced e2fsck run on the filesystem, and it should add the > indexes to all directories that lack it. It can also change t

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 10:06, Vincent Lefevre wrote: > > Without dir_index an ext filesystem with large directories is slow due > > to the linear nature of directories. But with dir_index it should be > > using a B-tree data structure and should be much faster. > > So, why is it slow? What ker

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Reco
On Tue, Apr 14, 2015 at 03:15:02PM +0200, Vincent Lefevre wrote: > On 2015-04-14 12:15:03 +0300, Reco wrote: > > On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: > > > On Tue, 14 Apr 2015 10:12:28 +0300 > > > Reco wrote: > > > > On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen w

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 08:47:15 -0300, Henrique de Moraes Holschuh wrote: > Also, some of the Unix best-practices do address this kind of > filesystem issue. One such best-practice is that you don't remove > just the files in ephemeral directories: you either use an ephemeral > filesystem (tmpfs) in the fir

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 12:15:03 +0300, Reco wrote: > On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: > > On Tue, 14 Apr 2015 10:12:28 +0300 > > Reco wrote: > > > On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: > > > Removing files from the directory does not change directory's i

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-13 16:28:27 -0600, Bob Proulx wrote: > Vincent Lefevre wrote: > > Loïc Grenié wrote: > > > Vincent Lefevre wrote: > > > > The problem is that this operation is (always?) very slow: something > > > > like 100 seconds (1 minute and 40 seconds). It has been reproducible > > > > for several

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-13 15:50:40 -0500, David Wright wrote: > That's staggering. My /var/lib/dpkg/info has ~8900 files and occupies > 462848 bytes. So that would be over ½million files in your case. > Does eftests stand for "excessive files tests"! It means "elementary function tests", but what this doesn't

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 04:22, Petter Adsen wrote: > But if you create new files in that directory after deleting them, I > expect the inodes get reallocated? They do, but a directory does not store inodes. http://en.wikipedia.org/wiki/Inode For ext4: https://ext4.wiki.kernel.org/index.php/Ext4_

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Mon, Apr 13, 2015, at 19:28, Bob Proulx wrote: > Without dir_index an ext filesystem with large directories is slow due > to the linear nature of directories. But with dir_index it should be > using a B-tree data structure and should be much faster. This can be > turned off for a migration if

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Gene Heskett
On Tuesday 14 April 2015 02:58:20 Petter Adsen wrote: > On Mon, 13 Apr 2015 15:41:03 +0200 > > Vincent Lefevre wrote: > > On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: > > > 2015-04-13 14:39 GMT+02:00 Vincent Lefevre : > > > > The problem is that this operation is (always?) very slow: > > > >

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Reco
Hi. On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: > On Tue, 14 Apr 2015 10:12:28 +0300 > Reco wrote: > > > Hi. > > > > On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: > > > On Mon, 13 Apr 2015 15:41:03 +0200 > > > Vincent Lefevre wrote: > > > > > > > On 2015-04

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Petter Adsen
On Tue, 14 Apr 2015 10:12:28 +0300 Reco wrote: > Hi. > > On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: > > On Mon, 13 Apr 2015 15:41:03 +0200 > > Vincent Lefevre wrote: > > > > > On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: > > > > 2015-04-13 14:39 GMT+02:00 Vincent Lefevr

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Reco
Hi. On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: > On Mon, 13 Apr 2015 15:41:03 +0200 > Vincent Lefevre wrote: > > > On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: > > > 2015-04-13 14:39 GMT+02:00 Vincent Lefevre : > > > > The problem is that this operation is (always?) very

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Petter Adsen
On Mon, 13 Apr 2015 15:41:03 +0200 Vincent Lefevre wrote: > On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: > > 2015-04-13 14:39 GMT+02:00 Vincent Lefevre : > > > The problem is that this operation is (always?) very slow: > > > something like 100 seconds (1 minute and 40 seconds). It has been >

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Bob Proulx
Vincent Lefevre wrote: > Loïc Grenié wrote: > > Vincent Lefevre wrote: > > > The problem is that this operation is (always?) very slow: something > > > like 100 seconds (1 minute and 40 seconds). It has been reproducible > > > for several months. The logs show nothing during this operation. > > > >

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Gene Heskett
On Monday 13 April 2015 16:08:44 Henrique de Moraes Holschuh wrote: > On Mon, Apr 13, 2015, at 13:48, Gene Heskett wrote: > > I wonder if our "sort" utilities also do that file shrink? > > No. It is something the VFS (generic filesystem layer inside the > kernel) or filesystem itself would have to

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): > Thanks! I didn't know that (I thought that the file system would > automatically "optimize" directories when files are removed, so > I've never looked closely at their size). Indeed: > > ypig:~/eftests> ls -ld tmp > drwxr-xr-x 2 vlefevre vlefevre 296

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Henrique de Moraes Holschuh
On Mon, Apr 13, 2015, at 13:48, Gene Heskett wrote: > I wonder if our "sort" utilities also do that file shrink? No. It is something the VFS (generic filesystem layer inside the kernel) or filesystem itself would have to do, due to atomicity requirements. > Unrelated of course, but of all the d

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Gene Heskett
On Monday 13 April 2015 09:41:03 Vincent Lefevre wrote: > On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: > > 2015-04-13 14:39 GMT+02:00 Vincent Lefevre : > > > The problem is that this operation is (always?) very slow: > > > something like 100 seconds (1 minute and 40 seconds). It has been > > >

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Vincent Lefevre
On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: > 2015-04-13 14:39 GMT+02:00 Vincent Lefevre : > > The problem is that this operation is (always?) very slow: something > > like 100 seconds (1 minute and 40 seconds). It has been reproducible > > for several months. The logs show nothing during this

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Darac Marjal
On Mon, Apr 13, 2015 at 02:39:40PM +0200, Vincent Lefevre wrote: > I have some regular directory ~/eftests/tmp, and after a reboot, > I often check that it is empty (as I remove its contents before > the reboot): from the ~/eftests directory, I do > > ypig:~/eftests> ll tmp > > which correspond

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Loïc Grenié
2015-04-13 14:39 GMT+02:00 Vincent Lefevre : > I have some regular directory ~/eftests/tmp, and after a reboot, > I often check that it is empty (as I remove its contents before > the reboot): from the ~/eftests directory, I do > > ypig:~/eftests> ll tmp > > which corresponds to > > ls -bF --co

reading an empty directory after reboot is very slow

2015-04-13 Thread Vincent Lefevre
I have some regular directory ~/eftests/tmp, and after a reboot, I often check that it is empty (as I remove its contents before the reboot): from the ~/eftests directory, I do ypig:~/eftests> ll tmp which corresponds to ls -bF --color -l tmp after alias expansion. The problem is that this