Add CONCURRENTLY option to REPACK When this flag is specified, REPACK no longer acquires access-exclusive lock while the new copy of the table is being created; instead, it creates the initial copy under share-update-exclusive lock only (same as vacuum, etc), and it follows an MVCC snapshot; it sets up a replication slot starting at that snapshot, and uses a concurrent background worker to do logical decoding starting at the snapshot to populate a stash of concurrent data changes. Those changes can then be re-applied to the new copy of the table just before swapping the relfilenodes. Applications can continue to access the original copy of the table normally until just before the swap, which is the only point at which the access-exclusive lock is needed.
There are some loose ends in this commit: 1. concurrent repack needs its own replication slot in order to apply logical decoding, which are a scarce resource and easy to run out of. 2. due to the way the historic snapshot is initially set up, only one REPACK process can be running at any one time on the whole system. 3. there's a danger of deadlocking (and thus abort) due to the lock upgrade required at the final phase. These issues will be addressed in upcoming commits. The design and most of the code are by Antonin Houska, heavily based on his own pg_squeeze third-party implementation. Author: Antonin Houska <[email protected]> Co-authored-by: Mihail Nikalayeu <[email protected]> Co-authored-by: Álvaro Herrera <[email protected]> Reviewed-by: Matthias van de Meent <[email protected]> Reviewed-by: Srinath Reddy Sadipiralla <[email protected]> Reviewed-by: Amit Kapila <[email protected]> Reviewed-by: Jim Jones <[email protected]> Reviewed-by: Robert Treat <[email protected]> Reviewed-by: Noriyoshi Shinoda <[email protected]> Reviewed-by: vignesh C <[email protected]> Discussion: https://postgr.es/m/5186.1706694913@antos Discussion: https://postgr.es/m/[email protected] Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/28d534e2ae0ac888b5460f977a10cd9bb017ef98 Modified Files -------------- doc/src/sgml/monitoring.sgml | 37 +- doc/src/sgml/mvcc.sgml | 12 +- doc/src/sgml/ref/repack.sgml | 118 +- src/Makefile | 1 + src/backend/access/heap/heapam.c | 26 +- src/backend/access/heap/heapam_handler.c | 290 +++- src/backend/access/heap/rewriteheap.c | 6 +- src/backend/catalog/system_views.sql | 19 +- src/backend/commands/Makefile | 1 + src/backend/commands/matview.c | 1 + src/backend/commands/meson.build | 1 + src/backend/commands/repack.c | 1835 ++++++++++++++++++-- src/backend/commands/repack_worker.c | 533 ++++++ src/backend/commands/tablecmds.c | 1 + src/backend/commands/vacuum.c | 12 +- src/backend/libpq/pqmq.c | 5 + src/backend/meson.build | 1 + src/backend/postmaster/bgworker.c | 5 + src/backend/replication/logical/decode.c | 28 +- src/backend/replication/pgrepack/Makefile | 32 + src/backend/replication/pgrepack/meson.build | 18 + src/backend/replication/pgrepack/pgrepack.c | 287 +++ src/backend/storage/ipc/procsignal.c | 3 + src/backend/tcop/postgres.c | 3 + src/backend/utils/activity/wait_event_names.txt | 1 + src/bin/psql/tab-complete.in.c | 4 +- src/include/access/heapam_xlog.h | 2 + src/include/access/tableam.h | 15 +- src/include/commands/progress.h | 17 +- src/include/commands/repack.h | 15 +- src/include/commands/repack_internal.h | 125 ++ src/include/replication/decode.h | 4 + src/include/storage/lockdefs.h | 4 +- src/include/storage/procsignal.h | 1 + src/makefiles/Makefile.cygwin | 2 + src/makefiles/Makefile.win32 | 2 + src/test/modules/injection_points/Makefile | 2 + .../modules/injection_points/expected/repack.out | 113 ++ .../injection_points/expected/repack_toast.out | 65 + src/test/modules/injection_points/meson.build | 2 + .../modules/injection_points/specs/repack.spec | 142 ++ .../injection_points/specs/repack_toast.spec | 112 ++ src/test/regress/expected/cluster.out | 8 + src/test/regress/expected/rules.out | 19 +- src/test/regress/sql/cluster.sql | 6 + src/tools/pgindent/typedefs.list | 6 + 46 files changed, 3682 insertions(+), 260 deletions(-)
