Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-16 Thread Dragan Milivojević
> You select only the Incremental job to restore, of course the data > is in one of the previous job. We can probably improve the error > message, or remove the meta-data-only record that is reported > as a file. > > I do not recommend to use the jobid= parameter in the restore > command line unles

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-16 Thread Eric Bollengier via Bacula-users
Hello Dragan, On 9/27/24 14:16, Dragan Milivojević wrote: Search the mailing list for a thread that I started a few days ago. It breaks bacula by saving empty files, no metadata, only name is saved. I have looked more closely, and your restore command is incorrect. You select only the Increme

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-15 Thread Eric Bollengier via Bacula-users
Hello Dragan, On 9/27/24 14:16, Dragan Milivojević wrote: Search the mailing list for a thread that I started a few days ago. It breaks bacula by saving empty files, no metadata, only name is saved. Did you open an issue on the bug tracker? gitlab.bacula.org Thanks Best Regards, Eric ___

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-11 Thread Marco Gaiarin
Mandi! Marco Gaiarin In chel di` si favelave... > No, i'm still working on snapshotting, next step backup; i've not clear if > Include{} and Exclude{} fileset directive apply to 'stippath' or > 'non-strippath' path... OK, things start working. The script do correctly snapshot; if needed, i can

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-09 Thread Marco Gaiarin
Mandi! Arno Lehmann In chel di` si favelave... > So I guess a good recommendation would be to make sure strippath is > applied to *all* paths in a file set, unless you are really sure you > know what you do. Uh, oh... clearly sure! > Also, did you per chance already try, in combination with

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-07 Thread Arno Lehmann
Hi Marco, Am 07.10.2024 um 14:50 schrieb Marco Gaiarin: Mandi! Gary R. Schmidt In chel di` si favelave... ... ...i don't think is about restoring, but backing up; indeed the example could be instead: if you have a fileset with an exactly. And with paths such as /usr/bin and /.snap/bin i

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-07 Thread Marco Gaiarin
Mandi! Gary R. Schmidt In chel di` si favelave... > Because it does exactly what it says it will do. > If you backup /a/b/c/d, and then restore it with "strippath=1" it will > restore to /b/c/d. > If you have some pre-existing /b/c/d it's now been over-written, or > overlaid, if you prefer..

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-05 Thread Gary R. Schmidt
On 05/10/2024 02:16, Marco Gaiarin wrote: [SNIP] > > Doc say: strippath=integer This option will cause integer paths to be stripped from the front of the full path/filename being backed up. This can be useful if you are migrating data from another vendor or if you have taken a snapshot

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-04 Thread Marco Gaiarin
Mandi! Arno Lehmann In chel di` si favelave... > Look at > https://www.bacula.org/15.0.x-manuals/en/main/Configuring_Director.html#SECTION002370 > > and search for "strippath" :-) Uh, oh! And was available also on 9.4! https://www.bacula.org/9.4.x-manuals/en/main/Con

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-03 Thread Arno Lehmann
Hi Marco, Am 02.10.2024 um 18:07 schrieb Marco Gaiarin: ... Initially i supposed Bacula will have some 'ignore prefix' parameters that will permit that, but this is available only on restore, not backup. For backup, data have to be 'rooted' (mounted) on the same path... Look at https://www.ba

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-02 Thread Marco Gaiarin
Mandi! Josip Deanovic via Bacula-users In chel di` si favelave... >> Bacula will just do it. Nothing special required. > This statement might be misleading in this particular case Marco > described. > Bacula will be able to run incremental backup but if the mountpoint is > changing every time

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-02 Thread Dan Langille
On Tue, Oct 1, 2024, at 3:42 AM, Josip Deanovic via Bacula-users wrote: > On 2024-10-01 03:16, Dan Langille wrote: >> On Mon, Sep 30, 2024, at 10:48 AM, Marco Gaiarin wrote: >>> Mandi! Dan Langille >>> In chel di` si favelave... >>> From https://dan.langille.org/2023/12/24/backing-up-

Re: [Bacula-users] Strategies for backup of millions of files...

2024-10-01 Thread Josip Deanovic via Bacula-users
On 2024-10-01 03:16, Dan Langille wrote: On Mon, Sep 30, 2024, at 10:48 AM, Marco Gaiarin wrote: Mandi! Dan Langille In chel di` si favelave... From https://dan.langille.org/2023/12/24/backing-up-freebsd-with-bacula-via-zfs-snapshot/ : I'm still doing some experimentation on this, indeed.

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-30 Thread Dan Langille
On Mon, Sep 30, 2024, at 10:48 AM, Marco Gaiarin wrote: > Mandi! Dan Langille > In chel di` si favelave... > >> From >> https://dan.langille.org/2023/12/24/backing-up-freebsd-with-bacula-via-zfs-snapshot/ >> : > > I'm still doing some experimentation on this, indeed. > > >> It doesn't all run a

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-30 Thread Marco Gaiarin
Mandi! Josip Deanovic via Bacula-users In chel di` si favelave... > Unless the "Ignore FileSet Change" option is used, Bacula will compare the MD5 > checksum of the Include/Exclude contents of the FileSet and decide whether a > Full backup is needed or not. > I don't recommend using this option

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-30 Thread Marco Gaiarin
Mandi! Dan Langille In chel di` si favelave... > From > https://dan.langille.org/2023/12/24/backing-up-freebsd-with-bacula-via-zfs-snapshot/ > : I'm still doing some experimentation on this, indeed. > It doesn't all run as incrementals. If the list of DATASETS (see above URL) > does not ch

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-28 Thread Dan Langille
On Thu, Sep 26, 2024, at 7:02 AM, Marco Gaiarin wrote: > Mandi! Arno Lehmann > In chel di` si favelave... > >> I have not looked up the discussion leading here. > > See my previous post: i'm fighting with ZFS. ;-) > > >> What I usually prefer is a combination of a Run Script on the client to >>

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-27 Thread Dragan Milivojević
Search the mailing list for a thread that I started a few days ago. It breaks bacula by saving empty files, no metadata, only name is saved. On Fri, 27 Sept 2024, 10:28 Josip Deanovic via Bacula-users, < bacula-users@lists.sourceforge.net> wrote: > On 2024-09-26 17:01, Dragan Milivojević wrote: >

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-27 Thread Josip Deanovic via Bacula-users
On 2024-09-26 17:01, Dragan Milivojević wrote: For example, while browsing/restoring backup of the specific date, you would get the state of a backed up directory as it was during the execution of a selected backup job. I would recommend the "accurate" option in this case. Be careful with the

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-26 Thread Dragan Milivojević
> For example, while browsing/restoring backup of the specific date, you > would > get the state of a backed up directory as it was during the execution of > a > selected backup job. > I would recommend the "accurate" option in this case. Be careful with the accurate option, if you use the recomme

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-26 Thread Josip Deanovic via Bacula-users
On 2024-09-26 13:02, Marco Gaiarin wrote: Hello Marco! Or it is still the 'fileset change' the trigger, so scripts can make all the dumbest things, but bacula keep going on incrementals? Also, if my script mount on '/some/dirs-202040926' and pass this as a backup dir for a full, but tomorrow

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-26 Thread Marco Gaiarin
Mandi! Arno Lehmann In chel di` si favelave... > I have not looked up the discussion leading here. See my previous post: i'm fighting with ZFS. ;-) > What I usually prefer is a combination of a Run Script on the client to > create a file list to back up, and a File= entry in the include list

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-15 Thread Arno Lehmann
Hi Marco, Am 13.09.2024 um 17:42 schrieb Marco Gaiarin: Mandi! Marco Gaiarin In chel di` si favelave... EG, i'll come back on this on mid-september... Still working on script. I have not looked up the discussion leading here. For a sake of mental health i want to mount snapshot on di

Re: [Bacula-users] Strategies for backup of millions of files...

2024-09-15 Thread Marco Gaiarin
Mandi! Marco Gaiarin In chel di` si favelave... > EG, i'll come back on this on mid-september... Still working on script. For a sake of mental health i want to mount snapshot on different mount point... so different file path. But path are defined in filesets, and i cannot define adifferent f

Re: [Bacula-users] Strategies for backup of millions of files...

2024-08-18 Thread Marco Gaiarin
Mandi! Dan Langille In chel di` si favelave... >> >> https://dan.langille.org/2023/12/24/backing-up-freebsd-with-bacula-via-zfs-snapshot/ > How is that working out? You don't know?! ;-) I'm currently on holiday; i'll come back next week, but i need to move and/or do some other tasks.

Re: [Bacula-users] Strategies for backup of millions of files...

2024-08-07 Thread Dan Langille
On Fri, Aug 2, 2024, at 11:03 AM, Marco Gaiarin wrote: > Mandi! Andrea Venturoli > In chel di` si favelave... > >> Probably yes, but why bother? >> Just take multiple snapshots: they have by default a different >> mountpoint and don't take any space. > > Right... never minded about that... i've

Re: [Bacula-users] Strategies for backup of millions of files...

2024-08-03 Thread Marco Gaiarin
Mandi! Andrea Venturoli In chel di` si favelave... > Probably yes, but why bother? > Just take multiple snapshots: they have by default a different > mountpoint and don't take any space. Right... never minded about that... i've also found some example: https://dan.langille.org/2023/1

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-29 Thread Andrea Venturoli
On 7/26/24 14:31, Marco Gaiarin wrote: Yes, FS is ZFS so i can snapshot them. But FS/mountpoint is the same for all data, and i've 5 jobs for spliting and optimizing them. There's some way to 'snapshot' FS only one time, with multiple jobs? Probably yes, but why bother? Just take multiple sn

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-26 Thread Marco Gaiarin
Mandi! Josip Deanovic via Bacula-users In chel di` si favelave... > Alternatively, Bacula has the noatime option which is not set by > default. I've give it a try. Seems it is useful, rather not dramatic. Thanks! -- ___ Bacula-users mailing l

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-26 Thread Marco Gaiarin
Mandi! Bill Arlofski via Bacula-users In chel di` si favelave... > The typical way to help with this type of situation is to create several > Fileset/Job pairs and then run them all > concurrently. Each Job would be reading a different set of directories. Finally we have splitted data using m

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-26 Thread Marco Gaiarin
Mandi! Dan Langille In chel di` si favelave... > Do your filesystem have any snapshot capabilities? With that many files, > backing up a snapshot would give you better results with respect to > consistency. Cool! Never minded about that! Yes, FS is ZFS so i can snapshot them. But FS/mountpo

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-17 Thread Josip Deanovic via Bacula-users
On 2024-07-15 17:26, Marco Gaiarin wrote: We have found that a dir (containing mostly home directories) with roughly one and a half million files, took too much time to be backud up; it is not a problem of backup media, also with spooling it took hours to prepare a spool. There's some strateg

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-16 Thread Dan Langille
On Mon, Jul 15, 2024, at 11:26 AM, Marco Gaiarin wrote: > For example, currently i have: > > Options { > Signature = MD5 > accurate = sm > } > > if i remove signature and check only size, i can gain some performance? You might. You also lose integrity checks - is the file I ju

Re: [Bacula-users] Strategies for backup of millions of files...

2024-07-15 Thread Bill Arlofski via Bacula-users
On 7/15/24 9:26 AM, Marco Gaiarin wrote: We have found that a dir (containing mostly home directories) with roughly one and a half million files, took too much time to be backud up; it is not a problem of backup media, also with spooling it took hours to prepare a spool. There's some strategy i

[Bacula-users] Strategies for backup of millions of files...

2024-07-15 Thread Marco Gaiarin
We have found that a dir (containing mostly home directories) with roughly one and a half million files, took too much time to be backud up; it is not a problem of backup media, also with spooling it took hours to prepare a spool. There's some strategy i can accomplish to reduce backup time (bac