On Thursday 10 February 2005 01:21 pm, David Busby <[EMAIL PROTECTED]> wrote: > I don't want this to turn into one of those ext3 vs xfs conversations. > > I want to know which filesystem is a better choice for having lots > (>4billion) directories. The dirs will be in tree format, so at the > root will be 256 dirs, each with 256 sub-dirs, each with 256 sub-dirs. > This will go on for 8 to 12 levels deep (I don't know yet) Then each > dir will have one small file in it (<32bytes).
Building a byte-wise Trie for fixed-length strings on a filesystem? BTW, if you have one layer that ~256 directories. If you have 2 layers that's ~(256 x 256) directories. So, with 8 layers that's roughly (256 ^ 8) = 2^8 ^ 8 = 2^64 ~= 16 quintillion (billion billion) directories. > How would I tune ext3 > for this? ext3 sucks; don't use it. Think of it as a reference implementation of the journaled unix file system. :P > What about XFS, I read that some of it's features might be > useful to me for this project. With XFS, it might be better to use large, structured files instead of lots of small files in many directories. ReiserFS is good at handling directories like this. Also, if you don't turn on notail (which will cost you some performance) you'll save a *lot* of space. With ext3 or XFS, each of your files wil end up taking a minimum of 512 bytes, unless they are completely empty. [AFAIK, in all filesystems except ReiserFS and NTFS a file must take up a whole number of 'blocks', no matter it's size; both ReiserFS and NTFS can stuff multiple files into a single block.] Note that with 2^32 directories (roughly 4 billion; which is only 4 or 5 levels deep in your arrangement) each containing a single file of 512 bytes (= 2^9 bytes), assuming NO directory overhead, that's 2^41 bytes (= 2 Terabytes). While there a many systems available with this kinda of space, do you have it? [ReriserFS/NTFS will shrink this down to (*best case*) 2^37 bytes (= 128 Gigabytes).] [ReiserFS does take an inordinately long time to *delete* a directory structure like this, but creating and searching it give good performance.] -- Boyd Stephen Smith Jr. [EMAIL PROTECTED] ICQ: 514984 YM/AIM: DaTwinkDaddy -- [email protected] mailing list
