OK, we still have the same issue here. The backup jobs are still running but the volumes are already in a much larger size than they should be.
I verified the sql query as well. Here the list is also empty. But interestingly this behavior is only present for one of the affected clients. For another one where the incrementals are also way too large, I'm able to get a list of files. For this client, I copied the list to the client and calculated the total file size, but it's not even close to the size of the related Bareos volume. So Bareos is definitely doing something wrong here. Note that we only use volumes once for a single job so the size can not be related to other jobs. One guess I had was that Bareos is somehow not keeping track of what's in the former incrementals and is including the same files in each incremental again and again. But the incrementals are still smaller than the full system, so Bareos is not backing up the full file system each time. The biggest affected system is around 8 TB large, but the backups are always around 1.15 TB, with the amount of files always being nearly identical with every job (805066, 811411, 810618). How likely is it that the database encoding would cause these problems? Approximately a month ago we shrank the Catalog database (according to https://docs.bareos.org/TasksAndConcepts/CatalogMaintenance.html#compacting-your-postgresql-database) because the PostgreSQL Autovacuum was not properly configured and the DB just kept growing more and more. This lead to the warning regarding the database encoding. But we haven't seen any issues since then. The weird incremental growth started last week. I'm currently waiting for one long-running consolidate job to finish but once that is done, I would dump the database and recreate it according to https://docs.bareos.org/IntroductionAndTutorial/InstallingBareos.html#id7 before adding the dump again. It looks like the consolidate job will keep running for at least another hour before I can "fix" the DB. If we could have a look at the system together in the meantime, that would be much appreciated. On Wednesday, 18 March 2026 at 17:42:29 UTC+1 Bruno Friedmann (bruno-at-bareos) wrote: > yes estimate level=incremental will only show what will be saved. > > for a given jobid you can also ask directly the database > in bconsole > > .sql query="select p.path || f.name from file f left join path p on > p.pathid=f.pathid where jobid=12345;" > > That should return something normally. > > On Wednesday, 18 March 2026 at 16:47:52 UTC+1 Patrick Hasler wrote: > >> >> > I'm not yet sure how reliable is a volume name like >> >> Created new Volume "Customer-A/AI_Inc_8263" in catalog. >> > Is that one file in /mnt/bareos/backup or dir/file ? >> >> It's an actual directory structure. In the end this will result in a >> sub-directory in /mnt/bareos/backup with the customer name ("Customer-A) >> and then the Bareos volume files in it. In this example AI_Inc_8263 is a >> file in the directory /mnt/bareos/backup/Customer-A. >> This has worked reliably over years and was the suggested setup by >> Andreas Rogge. For our use-case this is very helpful because Bareos does >> not have a feature for "grouping" hosts together and we need to be able to >> tell how much storage a customer is using. >> >> I would be very surprised if the issue is on the storage's side. The >> backups are properly written and the volumes have the exact file size that >> the WebUI and job log are reporting as well. It's just not clear what the >> actual content is and why they are so huge out of nowhere. >> >> For now I've started a new consolidate job which resulted in a successful >> virtual full backup for some of the affected hosts. So my hope would be >> that the next incremental which will trigger tonight will be of regular >> size. >> >> If that would not be the case, would it be an option to have a look at >> the setup together tomorrow? If the problem persist we would need to find a >> solution as soon as possible, because we are now in a situation where we >> cannot trust that Bareos is actually doing proper backups of our machines. >> It refuses to give us the content of what's being backed up and it's >> filling up our storage endpoints with huge data with unknown content. >> >> In any case, I'll let you guys know tomorrow what the status is. >> >> > if the incremental has everyday such number of changes you might want >> to check them with estimate listing >> >> The estimate command (estimate level=incremental >> job=Job_Customer-A_Daily_AlwaysIncremental_client-01 accurate=yes listing) >> gives me a proper file list. If `level=incremental` is specified, should it >> only print files that have changed and would be included in the next >> incremental job, or will it still give a full list of all files included in >> the related FileSet to that job? Because the output looks like it gives us >> the full file tree of that system. >> >> Here are the command outputs: >> *estimate level=incremental >> job=Job_Customer-A_Daily_AlwaysIncremental_client-01 accurate=yes listing >> ... (huge list of files) >> 2000 OK estimate files=1,523,195 bytes=2,095,857,085,860 >> >> >> >> >> On Wednesday, 18 March 2026 at 15:52:53 UTC+1 Bruno Friedmann >> (bruno-at-bareos) wrote: >> >> I hope you noticed that your database is using the wrong encoding and >> should be fixed as soon as possible >> > Warning: Encoding error for database "bareos". Wanted SQL_ASCII, got >> UTF8 >> Try to follow our documentation when you initialize the catalog. >> >> >> Due to the use of folder/file in volume names maybe you are required to >> quotes that when asking list files from volume and the webui can't ? >> >> I'm not yet sure how reliable is a volume name like >> > Created new Volume "Customer-A/AI_Inc_8263" in catalog. >> Is that one file in /mnt/bareos/backup or dir/file ? >> >> > Consolidation jobs keep their original start/end time. >> You mean consolidated jobs ;-) the consolidation job has its own >> start/end time ... >> >> if the incremental has everyday such number of changes you might want to >> check them with estimate listing >> >> >> On Wednesday, 18 March 2026 at 11:42:34 UTC+1 Patrick Hasler wrote: >> >> I've attached the job log for one of these jobs. Nothing unusual from >> what I can tell. >> >> >Are you sure you didn't get caught by the fact that after consolidation, >> history is rewritten, and older jobid get new files in due to the >> consolidation? >> >> These are not consolidation jobs, but the regular running backup jobs. >> Consolidation jobs would start with "Start Virtual Backup" in the >> joblog. This is also visible by the timestamps in the screenshots. >> Consolidation jobs keep their original start/end time. >> >> On Wednesday, 18 March 2026 at 11:32:24 UTC+1 Bruno Friedmann >> (bruno-at-bareos) wrote: >> >> You may want to check what the joblog contain. >> >> All my AI job are showing the files backup'ed when there's some, and also >> all my volumes show me jobid stored on. >> >> Are you sure you didn't get caught by the fact that after consolidation, >> history is rewritten, and older jobid get new files in due to the >> consolidation ? >> >> On Wednesday, 18 March 2026 at 11:27:35 UTC+1 Patrick Hasler wrote: >> >> Hi Bruno, >> >> Thank you for the quick reply! The list is actually empty: >> *list files jobid=93451 >> * >> >> I tried it for mutiple jobids. For older job with the regular file size >> the command is working as expected and I get a list of files. >> >> >> >> >> On Wednesday, 18 March 2026 at 11:17:18 UTC+1 Bruno Friedmann >> (bruno-at-bareos) wrote: >> >> You can see what files have been backup'ed by using >> bconsole >> list files jobid=12345 >> >> On Wednesday, 18 March 2026 at 10:16:57 UTC+1 Patrick Hasler wrote: >> >> We noticed a quiet urgent issue on some of our larger systems where we do >> backup for with Bareos. Suddenly every new incremental backup is huge and >> also ruffly has the same size every time (See screenshots below). >> Usually I would suspect something changing on the system (For example a >> DB backup dump cronjob) but as this affects multiple completely unrelated >> systems, this can be ruled out. There is no indication of any kind of large >> file changes on all the affected hosts, so the question is: What is Bareos >> actually backing up here? >> >> If I go to the restore section to check what files were actually backed >> up by the job, the "File selection" section just stays empty, so we don't >> have any idea what's in the backups. Due to there huge size we also cannot >> just restore them to the client to check their content and as we use >> encryption we cannot simply extract the data from the volumes on the >> Storage Daemon's side. >> >> There was no recent Bareos software upgrade, so I don't think that this >> is a new software bug. Also the affected clients use different versions of >> the File Daemon (24.0.7, 25.0.2), while the Director and Storage Daemon >> both use 25.0.2. >> >> What could be an explanation for this kind of behavior? I mentioned >> Always Incremental because I'm pretty sure that the problem is somehow >> related to it as the new huge incrementals started showing up after the >> most recent VirtualFull. But this is not the first VirtualFull, so I'm not >> sure why this behavior suddenly started.[image: >> bareos_huge_incrementals-1.png] >> >> [image: bareos_huge_incrementals-2.png] >> >> -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/bareos-users/d31f6709-aa19-4622-b5fc-e092f121e652n%40googlegroups.com.
