Predrag Punosevac <[email protected]> wrote: > Ingo Schwarze <[email protected]> wrote: > > > Hi Predrag, > > > > Hi Ingo, > > > Predrag Punosevac wrote on Fri, Jan 23, 2015 at 03:24:00PM -0500: > > > > > I was following this discussion with the great interest but without > > > intend to participate in it until today. > > > > > > Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last > > > night I received an e-mail from an angry user who could not upload files > > > (the upload will fail or upload the file with file size zero). After > > > running df I noticed that /tmp was 100% full (4GB used) but the size of > > > individual files was only 12Kb. > > > > That is unlikely to be due to softdep. The usual reason for a file > > system to be actually full and seemingly almost empty at the same > > time is somebody doing the following sequence of operations: > > > > Now it seems obvious that I made a mistake and put the blame on metadata > instead of running fstat but at the moment I had a mindset of DF user. > Namely couple of months ago I had a rogue process which keeps altering > log files on one of DF machines. Long story short after couple of hours > my /var had only about 1GB of log files but HAMMER history almost > completely filled the rest of /var due to the frequent changes of log > files. I regained the space by clearing HAMMER history and learned how > to tune HAMMER retention parameters. > > It seems from what you are saying and from what seems clear to me before > last night incident that I made a fundamental mistake of thinking of > soft updates as honest journaling file system while in reality they are > just a way of maintaining file system meta-data integrity. Hence softdep > are safe from HAMMER like history retention problems. > > Most Kind Regards, > Predrag > > P.S. Since you are developer I can't resist not to ask if anybody is > looking at Bitrig code which brings WAPBL essentially into OpenBSD? > > > > - open(2) a file for writing > > - writing a lot of data until the file system is full > > - unlink(2) the file > > - keep the process running that open(2)'ed it > > - let that process keep the file open and *not* close(2) it > > > > Specifically, in /tmp, anybody can do that. > > > > > I thought for a second and I remember seeing this with HAMMER on DF. > > > > The above works with almost any file system. > > > > > Long story short I checked /etc/fstab and > > > sure enough I had rw,softdep next to each partition including tmp. I > > > removed softdep rebooted the sytem and /tmp usage dropped to 0%. > > > > That's not likely to be related to softdep either. Chances are > > just rebooting would have solved the issue as well - simply because > > rebooting terminates all running processes, and consequently closes > > all open files. > > > > What you should have done instead was run fstat(1), look for processes > > having files open in /tmp, use ls(1) -iRa /tmp to find the inode > > numbers of linked files in /tmp, and kill the processes having files > > open that were *not* linked until you found the one having the big > > file open - and then have a friendly talk with the responsible user, > > if any, or figure out what went wrong in case some daemon process > > caused the issue. > >
Dear Ingo, Sure enough it happen again even without softdep. This time I followed your instructions closely. Apache which runs in the insecure mode because of Mollify was the culprit. /tmp was filled with some php garbage. Restarting Apache and removing php files and /tmp was 0% again. Most Kind Regards, Predrag > > > My questions is which partitions should be mounted with softdep > > > option? > > > > I'm not an expert on that and hardly ever use softdep, but i'd say > > on file systems where file create/delete performance is *critically* > > important, performace has been clearly demonstrated to be insufficient > > without softdep, and you consider data loss harmless. > > > > > Is there a way to prune metadata which will save me for problems like > > > the one I encountered last night. > > > > I'm not convinced that's a good question to ask. > > > > Yours, > > Ingo
