Kevin Fishburne ha scritto: >>> My current plan is to create a directory for each region >>> ([65536/32/32]^2). Each region directory contains 32^2 data files >>> (1024). Hopefully this won't stress any particular file system as far as >>> how many directories and files are contained within a single directory. >>> >>> >> But... I am missing something... the number of files was 4M, right? And >> 64 directories with 1024 files does not sum up to 4M... >> > > A cell is 32x32 tiles (bytes), and the map is 65536^2 tiles, so there > are 2048^2 cells. Each of these are organized into directories of 32^2 > cells (regions). The data files alone are 4.2 million in number, the > directories that organize them hopefully add a layer of filesystem > efficiency. Right now a separate directory is created for each region > and empty data files are created for each cell within that region. > > I can't explain myself why I did not understand at first. Well (not)... apparently I missed the square "^2".
So there are 4096 directories containing 1024 files each, and each file is 1Kb in size (or not?). If this is correct, you should choose a small allocation unit for your file system (1 or 2 Kb?). Assuming that the file system uses, say, 128 bytes for storing the metadata for a file, you will have 4Gb of data, 512Mb of metadata for the files, and 4K*128=512K of metadata for the directories. More than 12% of the total volume of data is spent for organizing them, which is not a bad number, but in your case this could be improved by merging together, for example, all the files (1024) of a single directory, if this is possible. If the files were different in length, then using the file system to organize them would have been the easiest way. But this is just a theoretical supposition... I have checked the file systems on my old server. The root partition is 2Gb (your cells would not fit there), uses 1K blocks, and has a total of 1M inodes (again the cells would not fit). The /usr partition is 18Gb, uses a block size of 4K, and has a total of 1M inodes. Again, your data would not fit because of the number of files. The biggest partition is 200Gb, uses 4K as block size, and counts 12M inodes. In this partition, your data would take 16Gb of space, but anyway it would fit, at the cost of transferring 4 times the useful data at every read (probably). I think the only pit is speed, and not disk space wasting. If there is enough ram in the computer to cache the 512Mb of metadata, access is very quick (10 milliseconds? Or less?). Otherwise, the same metadata must be read over and over, at every access, which means that the computer will read several Mb to access a single 1k file. These are the opposite and extreme situations - in the reality a certain amount of disk cache will always be in use. My old server has only 96Mb of ram, but as a normal file server performs not badly. Clearly, I don't have 4M files on it... :-) Regards, Doriano ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Gambas-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/gambas-user
