Re: inodes usage of svn server
Bo Berglund wrote on Wed, 03 Jan 2018 07:50 +0100: > Is this a one-time cleanup operation or does it need to be executed > regularly? Regularly. > I.e. does running "svnadmin pack" on the repo consolidate > the many files into a smaller number of big files and set some > repository property such that it will work that way in the future? The former. > This is what I found in svnbook 1.7 page 178: > > "By concatenating all the files of a completed > shard into a single “pack” file and then removing the original > per-revision files, svnadmin pack reduces the file count within a > given shard down to just a single file." > > and > > "Repacking packed shards is legal, but will have no effect on the disk > usage of the repository." > > What exactly is a "shard"? > Is it one of the numerical directories each containing exactly 1000 > files? > Yes. > If so the "shard" storage on my repos seem to only cost at most 4% > extra disk space compared to using a single file (comparing file sizes > against disk usage). The concern was inode usage, not disk space usage. Packing reduces the number of inodes by 99.8%. > Or will compression of the "svnadmin pack" operation make the combined > size of the files even smaller? I.e. it does not only save unused file > allocation space but actually packs the content even better? 'svnadmin pack' neither recompresses nor redeltifies. It just concatenates the files.
Re: inodes usage of svn server
On Wed, Jan 3, 2018 at 1:50 AM, Bo Berglund wrote: > On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia > wrote: > >>On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <356730...@qq.com> wrote: >>> >>> inodes usage of my svn server grows up quickly as repositories commits >>> increasing. >>> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+. >> >>There is "svnadmin pack", which needs to run on the Subversion >>repository server. >> > > Is this a one-time cleanup operation or does it need to be executed > regularly? I.e. does running "svnadmin pack" on the repo consolidate > the many files into a smaller number of big files and set some > repository property such that it will work that way in the future? > > This is what I found in svnbook 1.7 page 178: > > "By concatenating all the files of a completed > shard into a single “pack” file and then removing the original > per-revision files, svnadmin pack reduces the file count within a > given shard down to just a single file." > > and Disk space != inodes. You asked about inodes. File systems have a maximum number of inodes, the maximum number of files directories, links, etc. that they know how to count for that filesystem at the time the filesystem is built. Storing information about files takes space on a filesystem, sot it's notrmally pre-allocated when making the filesystem. From your own description, you've run out of inodes and need to free some up, or rebuild your environment to better handle structures that handle many small files. > "Repacking packed shards is legal, but will have no effect on the disk > usage of the repository." > > What exactly is a "shard"? > Is it one of the numerical directories each containing exactly 1000 > files? It's one of the files in those directories. > If so the "shard" storage on my repos seem to only cost at most 4% > extra disk space compared to using a single file (comparing file sizes > against disk usage). > Or will compression of the "svnadmin pack" operation make the combined > size of the files even smaller? I.e. it does not only save unused file > allocation space but actually packs the content even better? Please stop confusing "inode" with "disk space" It's more like you've run out of boxes to put things in, i.e. inodes, than running out of space to store store the boxes (i.e., disk space). > -- > Bo Berglund > Developer in Sweden >
Re: inodes usage of svn server
On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia wrote: >Please stop confusing "inode" with "disk space" It's more like you've >run out of boxes to put things in, i.e. inodes, than running out of >space to store store the boxes (i.e., disk space). Please do not confuse me with the original poster "Keva-Slient" I merely stepped in to clarify if I might also have the problem, which he apparently is having. But I do not think so (I am on Windows Server 2016 with an NTFS file system). -- Bo Berglund Developer in Sweden
Re: inodes usage of svn server
On Wed, Jan 3, 2018 at 11:54 AM, Bo Berglund wrote: > On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia > wrote: > >>Please stop confusing "inode" with "disk space" It's more like you've >>run out of boxes to put things in, i.e. inodes, than running out of >>space to store store the boxes (i.e., disk space). > > Please do not confuse me with the original poster "Keva-Slient" > I merely stepped in to clarify if I might also have the problem, which > he apparently is having. > But I do not think so (I am on Windows Server 2016 with an NTFS file > system). Sorry about the quotation confusion. NTFS also has limits on the maximum number of files for a filesystem. There is no complete escape from the underlying limitations of trade-offs of easy processing versus performance versus capacity that lead to these limitations. There are also notable performance limits on having too many files in a directory. We could have a long discussion about the fundamentals: suffice to say that organizing a bunch of associated bits into a single file, rather than scattering many such bits around a filesystem and overpopulating a directory or a filesystem is not your friend. > -- > Bo Berglund > Developer in Sweden >
Re: inodes usage of svn server
On 1/3/2018 13:19, Nico Kadel-Garcia wrote: NTFS also has limits on the maximum number of files for a filesystem. FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx) There are also notable performance limits on having too many files in a directory. I don't know enough about NTFS internals to say whether that's the case with NTFS or not, but in the context of this discussion, the default SVN shard size is 1000 revisions, which I don't think could be considered "too many files in a directory". While I don't have any actual numbers, my gut feeling is that packing the repo isn't really needed on NTFS in terms of day to day performance of clients using the repo, or in terms of filesystem limitations. That said, I do it anyway because backing up the repo does go faster when dealing with a couple dozen large files vs. tens of thousands of small files. -- Name: Dave Huang | Mammal, mammal / their names are called / INet: k...@azeotrope.org | they raise a paw / the bat, the cat / Telegram: @dahanc| dolphin and dog / koala bear and hog -- TMBG Dahan: Hani G Y+C 42 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+ PL++
Re: inodes usage of svn server
On 03.01.2018 21:13, Dave Huang wrote: > On 1/3/2018 13:19, Nico Kadel-Garcia wrote: >> NTFS also has limits on the maximum number of files for a filesystem. > > FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see > Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx) > >> There are also notable performance limits on having too many files in >> a directory. > > I don't know enough about NTFS internals to say whether that's the > case with NTFS or not, but in the context of this discussion, the > default SVN shard size is 1000 revisions, which I don't think could be > considered "too many files in a directory". > > While I don't have any actual numbers, my gut feeling is that packing > the repo isn't really needed on NTFS in terms of day to day > performance of clients using the repo, or in terms of filesystem > limitations. That said, I do it anyway because backing up the repo > does go faster when dealing with a couple dozen large files vs. tens > of thousands of small files. And there's your answer to the question of day to day performance of the repository: Subversion also has to open many vs. just one file when its reading historical revisions, so packing will definitely help to reduce the number of directory lookups and file opens (the latter are notoriously slow on Windows). Depending on usage patterns, the performance boost may be significant. -- Brane
Re: inodes usage of svn server
On Wed, 3 Jan 2018 21:16:02 +0100, Branko ?ibej wrote: >And there's your answer to the question of day to day performance of the >repository: Subversion also has to open many vs. just one file when its >reading historical revisions, so packing will definitely help to reduce >the number of directory lookups and file opens (the latter are >notoriously slow on Windows). Depending on usage patterns, the >performance boost may be significant. THat seems to be a valid cause for using the function then! I will look at my new converted repository in view of this info. Thanks! -- Bo Berglund Developer in Sweden
Re: inodes usage of svn server
On 03.01.2018 22:04, Bo Berglund wrote: > On Wed, 3 Jan 2018 21:16:02 +0100, Branko ?ibej > wrote: > >> And there's your answer to the question of day to day performance of the >> repository: Subversion also has to open many vs. just one file when its >> reading historical revisions, so packing will definitely help to reduce >> the number of directory lookups and file opens (the latter are >> notoriously slow on Windows). Depending on usage patterns, the >> performance boost may be significant. > THat seems to be a valid cause for using the function then! > I will look at my new converted repository in view of this info. > Thanks! It's safe to create a post-commit hook that runs 'svnadmin pack' asynchrouously, or schedule a task that does that (hourly or daily, depending on your needs). The operation is cheap if it has nothing to do. -- Brane
inode usage: when to delete transactions files after committing failed
there are many transactions files after committing failed in my svn server. they cost lots of inodes. when and how can i delete them? i've found out that server will generate new transaction dir xx/db/transactions/[0-9]+\-[0-9]+.txn dir after each failed committing. any tool can delete them when server running? i dont want to delete the txn dir which is in using. -- . Vanity...My favourite sin. --<>
