I just managed to get CI to pass. All I had to do was to move two suites to the top of the greenplum schedule: https://github.com/edespino/cloudberry/actions/runs/18274708119
-=e Ed Espino 925.389.4640 On Mon, Oct 6, 2025 at 1:02 AM Leonid Borchuk <[email protected]> wrote: > Hi, folk! > > I tried to investigate the issue too. Anyway, I can't finish PR without > tests. > > What I see: > > 1. There is no "evil" file or commit that broke the tests. What we stumbled > upon - Out of space. I saw some generated coredump files in tests, but they > also seemed to be a consequence of space exhaustion. > > df at the end of tests: > > Show disk usage info > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:1015 > >Filesystem > Type Size Used Avail Use% Mounted on > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:1016 > >overlay > overlay 73G 73G 99M 100% / > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:1017 > >tmpfs > tmpfs 64M 0 64M 0% /dev > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:1018 > >shm > tmpfs 2.0G 68K 2.0G 1% /dev/shm > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:1019 > >/dev/root > ext4 73G 73G 99M 100% /__w > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:1020 > >tmpfs > tmpfs 3.2G 9.2M 3.2G 1% /run/docker.sock > > 2. The top space consumer is cloudberry (One could find debug output in my > repo > > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548 > ). > All looks like we had been growing a little bit with each additional test, > and at the end has reached the limit of space. > > sudo du -c / | sort -n | tail -150 shows: > > 621320 /usr/lib64 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2459 > >655388 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1/pg_wal > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2460 > >655388 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2/pg_wal > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2461 > >655392 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast2/demoDataDir1/pg_wal > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2462 > >655396 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2/pg_wal > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2463 > >663736 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/standby/base > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2464 > >664192 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/base > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2465 > >720932 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/pg_wal > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2466 > >720936 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/pg_wal > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2467 > >856204 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2/base/17018 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2468 > >856996 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2/base/17018 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2469 > >904508 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/pg_distributedlog > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2470 > >910092 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/pg_distributedlog > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2471 > >984940 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1/base/17018 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2472 > >985876 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast2/demoDataDir1/base/17018 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2473 > >1049388 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/base/17018 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2474 > >1049444 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/base/17018 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2475 > >1225744 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2/base > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2476 > >1226736 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2/base > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2477 > >1277720 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/standby > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2478 > >1354452 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1/base > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2479 > >1355584 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast2/demoDataDir1/base > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2480 > >1420728 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0/base > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2481 > >1420796 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/base > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2482 > >1639176 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/pg_subtrans > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2483 > >1888592 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror3/demoDataDir2 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2484 > >1888596 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror3 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2485 > >1893504 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast3/demoDataDir2 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2486 > >1893508 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast3 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2487 > >1904984 > /usr > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2488 > >2017300 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror2/demoDataDir1 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2489 > >2017304 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror2 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2490 > >2023864 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast2/demoDataDir1 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2491 > >2023868 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast2 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2492 > >2549268 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0/pg_subtrans > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2493 > >3049412 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2494 > >3049416 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/qddir > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2495 > >3133184 > > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror1/demoDataDir0 > > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2496 > >3133188 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast_mirror1 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2497 > >5735508 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1/demoDataDir0 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2498 > >5735512 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs/dbfast1 > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2499 > >21019248 > /__w/cloudberry/cloudberry/gpAux/gpdemo/datadirs > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2500 > >21019320 > /__w/cloudberry/cloudberry/gpAux/gpdemo > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2501 > >21020984 > /__w/cloudberry/cloudberry/gpAux > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2502 > >22435396 > /__w/cloudberry/cloudberry > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2503 > >22435400 > /__w/cloudberry > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2504 > >22454024 > /__w > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2505 > >25110408 > / > < > https://github.com/open-gpdb/cloudberry/actions/runs/18248898786/job/51961002548#step:14:2506 > >25110408 > total > > Some observations from here: > > A. The total space size of one test is ~25Gb > B. pg_subtrans on segments ~ 2,5 Gb and more then data size > C. pg_distributedlog on segments ~ 1 Gb > D. pg_wal on segments ~700 mb > > So the envelope math here is: > > I. We have 3 heavy tests: ic-good-opt-on, ic-good-opt-off, ic-cbdb-parallel > II. All tests executed in parallel > III. Each test have 3 segments and 3 mirrors > IV. Total needed space size for tests 3 (the number of parallel tests) x > (pg_wal_size * 6 + pg_subtrans_size * 4 + pg_distributedlog * 2 ), which is > 3 * (0,7 * 6 + 2,5 * 4 + 1 * 2) = 48,6 Gb > > My thought and questions from here: > > 1. Could we set max-parallel in strategy to 2 (this will lengthen the > tests) > ? > 2. Could we set archive_command to /bin/true and do not store WAL files? > 3. How to understand why pg_subtrans is so big? There should be a long > transaction + a lot of subtransactions (savepoints?) - but 2,5 Gb ... > > On Sat, Oct 4, 2025 at 9:54 PM Ed Espino <[email protected]> wrote: > > > I have an updated mechanism to free unused space in the test > environments. > > Unfortunately, this is not resolving the testing issues. I will be > > attempting to isolate the issue to any recent code and CI changes. > > Additionally after a conversation with Tushar, I will be reaching out to > > the Apache Infrastructure team to identify necessary steps to use larger > CI > > resources (if possible). > > > > Stay tuned, > > -=e > > > > On Fri, Oct 3, 2025 at 2:50 AM Dianjin Wang <[email protected]> > wrote: > > > > > Cool, thanks Ed! > > > > > > > > > > > > Best, > > > Dianjin Wang > > > > > > > > > Ed Espino <[email protected]>于2025年10月3日 周五17:08写道： > > > > > > > I have determined that the test container is running out of disk > space > > > and > > > > this is leading to the testing issues. I am trying to determine if it > > is > > > > possible to clean up unused artifacts in the test container prior to > > test > > > > execution. > > > > > > > > -=e > > > > > > > > On Thu, Oct 2, 2025 at 9:51 PM Ed Espino <[email protected]> wrote: > > > > > > > > > I'll take a look. > > > > > > > > > > -=e > > > > > > > > > > -- > > > > > Ed Espino > > > > > Apache Cloudberry (Incubating) & MADlib > > > > > > > > > > On Thu, Oct 2, 2025 at 8:50 PM Dianjin Wang <[email protected] > > > > > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> I’m wondering if the CI might be having issues. On my PR #1358, > the > > > > >> jobs `ic-good-opt-off`, `ic-good-opt-on`, and `ic-cbdb-parallel` > > have > > > > >> been failing consistently, even after multiple reruns. > > > > >> > > > > >> I also noticed similar failures happening on other PRs. Could > > someone > > > > >> help check if the CI is currently down or unstable? > > > > >> > > > > >> > > > > >> Best, > > > > >> Dianjin Wang > > > > >> > > > > >> > > --------------------------------------------------------------------- > > > > >> To unsubscribe, e-mail: [email protected] > > > > >> For additional commands, e-mail: [email protected] > > > > >> > > > > >> > > > > > > > > > > > > > -- > > Ed Espino > > Apache Cloudberry (Incubating) & MADlib > > >
