Hello,
On 2/12/2007 11:43 AM, Daniel Holtkamp wrote:
> Hi !
>
> My bacula 2.0.1 installation is running quite nicely except for some
> servers. I`ll use only one of these as an example as the others have the
> same problem.
>
> This one server has to backup more than 5 million files that are very
> small (usually less than 2KB). The problem is that the performance
> impact backing up these files is enormous.
>
> Here is a little sniplet from the last (unfinished) backup.
>
> Elapsed time: 23 hours 51 mins 42 secs
> Priority: 10
> FD Files Written: 3,562,070
> SD Files Written: 3,561,858
> FD Bytes Written: 2,507,509,039 (2.507 GB)
> SD Bytes Written: 3,088,552,545 (3.088 GB)
> Rate: 29.2 KB/s
>
> At that time the backup ran for almost a complete day and it still has
> to backup 2+ million files that make up for about 3 GB of data. As you
> can see the rate is VERY slow. I have of course enabled attribute
> spooling to take the database out of the equation. Also the backup goes
> to diskbased-volumes. It only gets this slow when it gets to the loads
> of small files - prior to that the backup rate is perfectly acceptable
> with 2MB/s.
Such a number of tiny files is usually a problem. There are several
reasons to this, IMO: Disk seeks (often 2 per file: read inode, read
data) which is hard to avoid
Other possible limitations on backup throughput can be minmized, I hope:
> The fileset for this server is this:
>
> FileSet {
> Name = "X400mta"
> Include {
> Options {
> exclude = yes
> wilddir = "/var/tmp"
> regexdir = "/var/[cache/man|catman]/[cat?|X11R6/cat?|local/cat?]"
Probably a probelm. You could try to expand the directories in the
configuration.
> compression=GZIP
Try running the job without compression. You could even check if
compression matters much with this special fileset.
> signature=SHA1
This one might be the limiting factor: SHA1 means lots of CPU work.
Depending on the data you store, you could perhaps run this fileset
without computing signatures, or use the less cpu-intensive MD5 alternative.
> }
> File = /
> File = /opt
> File = /usr
> File = /var
> File = /export/home
> }
> Include {
> Options {
> regexdir = "/var/[cache/man|catman]/[cat?|X11R6/cat?|local/cat?]"
> keepatime=yes
> mtimeonly=yes
> compression=GZIP
> signature=SHA1
> }
> File = /var/tmp
> }
> Exclude {
> File = .autofsck
> File = /proc
> File = /tmp
> File = .journal
> File = /opt/rsi/archive
> File = /opt/rsi/spool
> File = /opt/x400/mtadata/logfiles
> }
> }
>
> Any ideas on how to improve performance here ? Can the excludes be a
> problem ? Or the Regex ?
The regex might be a problem, but I'd start with compression and
signatures first... both can be quite important, so if these are what
makes your backups slow you've got to choose...
> Also what influences the performance on migrating data ?
I have no idea whatsoever... except that I would observe the systems
load when running migration jobs. Not only the load itself, but also i/o
wait times and memory usage.
Arno
> I`ve had migration processes running nicely at 15MB/s (max for
> tapedrive) and some go at a measily 1 MB/s - from the same disk-array to
> the same tapedrive of course.
>
> Best regards,
> Daniel Holtkamp
>
>
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier.
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Bacula-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/bacula-users
--
IT-Service Lehmann [EMAIL PROTECTED]
Arno Lehmann http://www.its-lehmann.de
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users
