diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index a09ceb2..2f72633 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -5984,12 +5984,15 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; - VACUUM performs a whole-table scan if the table's + VACUUM performs an aggressive scan if the table's pg_class.relfrozenxid field has reached - the age specified by this setting. The default is 150 million - transactions. Although users can set this value anywhere from zero to - two billions, VACUUM will silently limit the effective value - to 95% of , so that a + the age specified by this setting. An aggressive scan differs from + a regular VACUUM in that it visits every page that might + contain unfrozen XIDs or MXIDs, not just those that might contain dead + tuples. The default is 150 million transactions. Although users can + set this value anywhere from zero to two billions, VACUUM + will silently limit the effective value to 95% of + , so that a periodical manual VACUUM has a chance to run before an anti-wraparound autovacuum is launched for the table. For more information see @@ -6028,9 +6031,12 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; - VACUUM performs a whole-table scan if the table's + VACUUM performs an aggressive scan if the table's pg_class.relminmxid field has reached - the age specified by this setting. The default is 150 million multixacts. + the age specified by this setting. An aggressive scan differs from + a regular VACUUM in that it visits every page that might + contain unfrozen XIDs or MXIDs, not just those that might contain dead + tuples. The default is 150 million multixacts. Although users can set this value anywhere from zero to two billions, VACUUM will silently limit the effective value to 95% of , so that a diff --git a/doc/src/sgml/maintenance.sgml b/doc/src/sgml/maintenance.sgml index 5204b34..a03c2c6 100644 --- a/doc/src/sgml/maintenance.sgml +++ b/doc/src/sgml/maintenance.sgml @@ -438,27 +438,32 @@ - VACUUM normally skips pages that don't have any dead row - versions, but those pages might still have row versions with old XID - values. To ensure all old row versions have been frozen, a - scan of the whole table is needed. - controls when - VACUUM does that: a whole table sweep is forced if - the table hasn't been fully scanned for vacuum_freeze_table_age - minus vacuum_freeze_min_age transactions. Setting it to 0 - forces VACUUM to always scan all pages, effectively ignoring - the visibility map. + VACUUM uses the visibility map + to determine which pages of a relation must be scanned. Normally, it + will skips pages that don't have any dead row versions even if those pages + might still have row versions with old XID values. Therefore, normal + scans won't succeed in freezing every row version in the table. + Periodically, VACUUM will perform an aggressive + vacuum, skipping only those pages which contain neither dead rows nor + any unfrozen XID or MXID values. + + controls when VACUUM does that: all-visible but not all-frozen + pages are scanned if the number of transactions that have passed since the + last such scan is greater than vacuum_freeze_table_age minus + vacuum_freeze_min_age. Setting + vacuum_freeze_table_age to 0 forces VACUUM to + use this more aggressive strategy for all scans. The maximum time that a table can go unvacuumed is two billion transactions minus the vacuum_freeze_min_age value at - the time VACUUM last scanned the whole table. If it were to go + the time of the last aggressive vacuum. If it were to go unvacuumed for longer than that, data loss could result. To ensure that this does not happen, autovacuum is invoked on any table that might contain unfrozen rows with XIDs older than the age specified by the configuration parameter . (This will happen even if + linkend="guc-autovacuum-freeze-max-age">. (This will happen even if autovacuum is disabled.) @@ -491,7 +496,7 @@ normal delete and update activity is run in that window. Setting it too close could lead to anti-wraparound autovacuums, even though the table was recently vacuumed to reclaim space, whereas lower values lead to more - frequent whole-table scans. + frequent aggressive vacuuming. @@ -527,7 +532,7 @@ pg_database. In particular, the relfrozenxid column of a table's pg_class row contains the freeze cutoff XID that was used - by the last whole-table VACUUM for that table. All rows + by the last aggressive VACUUM for that table. All rows inserted by transactions with XIDs older than this cutoff XID are guaranteed to have been frozen. Similarly, the datfrozenxid column of a database's @@ -554,18 +559,21 @@ SELECT datname, age(datfrozenxid) FROM pg_database; VACUUM normally only scans pages that have been modified since the last vacuum, but - relfrozenxid can only be advanced when the whole table is - scanned. The whole table is scanned when relfrozenxid is - more than vacuum_freeze_table_age transactions old, when - VACUUM's FREEZE option is used, or when all pages - happen to + relfrozenxid can only be advanced every page of the table + that might contain unfrozen XIDs is scanned. This happens when + relfrozenxid is more than + vacuum_freeze_table_age transactions old, when + VACUUM's FREEZE option is used, or when all + pages that are not already all-frozen happen to require vacuuming to remove dead row versions. When VACUUM - scans the whole table, after it's finished age(relfrozenxid) - should be a little more than the vacuum_freeze_min_age setting - that was used (more by the number of transactions started since the - VACUUM started). If no whole-table-scanning VACUUM - is issued on the table until autovacuum_freeze_max_age is - reached, an autovacuum will soon be forced for the table. + scans every page in the table that is not already all-frozen, it should + set age(relfrozenxid) to a value just a little more than the + vacuum_freeze_min_age setting + that was used (more by the number of transcations started since the + VACUUM started). If no relfrozenxid-advancing + VACUUM is issued on the table until + autovacuum_freeze_max_age is reached, an autovacuum will soon + be forced for the table. @@ -634,21 +642,23 @@ HINT: Stop the postmaster and vacuum that database in single-user mode. - During a VACUUM table scan, either partial or of the whole - table, any multixact ID older than + Whenever VACUUM scans any part of a table, it will replace + any multixact ID it encounters which is older than - is replaced by a different value, which can be the zero value, a single + by a different value, which can be the zero value, a single transaction ID, or a newer multixact ID. For each table, pg_class.relminmxid stores the oldest possible multixact ID still appearing in any tuple of that table. If this value is older than - , a whole-table - scan is forced. mxid_age() can be used on + , an aggressive + vacuum is forced. As discussed in the previous section, an aggressive + vacuum means that only those pages which are known to be all-frozen will + be skipped. mxid_age() can be used on pg_class.relminmxid to find its age. - Whole-table VACUUM scans, regardless of + Aggressive VACUUM scans, regardless of what causes them, enable advancing the value for that table. Eventually, as all tables in all databases are scanned and their oldest multixact values are advanced, on-disk storage for older @@ -656,13 +666,13 @@ HINT: Stop the postmaster and vacuum that database in single-user mode. - As a safety device, a whole-table vacuum scan will occur for any table + As a safety device, an aggressive vacuum scan will occur for any table whose multixact-age is greater than - . Whole-table + . Aggressive vacuum scans will also occur progressively for all tables, starting with those that have the oldest multixact-age, if the amount of used member storage space exceeds the amount 50% of the addressable storage space. - Both of these kinds of whole-table scans will occur even if autovacuum is + Both of these kinds of aggressive scans will occur even if autovacuum is nominally disabled. @@ -743,9 +753,9 @@ vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuple UPDATE and DELETE operation. (It is only semi-accurate because some information might be lost under heavy load.) If the relfrozenxid value of the table is more - than vacuum_freeze_table_age transactions old, the whole - table is scanned to freeze old tuples and advance - relfrozenxid, otherwise only pages that have been modified + than vacuum_freeze_table_age transactions old, an aggressive + vacuum is performed to freeze old tuples and advance + relfrozenxid; otherwise, only pages that have been modified since the last vacuum are scanned. diff --git a/src/backend/commands/vacuumlazy.c b/src/backend/commands/vacuumlazy.c index 363b2d0..401e218 100644 --- a/src/backend/commands/vacuumlazy.c +++ b/src/backend/commands/vacuumlazy.c @@ -106,6 +106,7 @@ typedef struct LVRelStats BlockNumber rel_pages; /* total number of pages */ BlockNumber scanned_pages; /* number of pages we examined */ BlockNumber pinskipped_pages; /* # of pages we skipped due to a pin */ + BlockNumber vmskipped_frozen_pages; /* # of pages we skipped by all-frozen bit */ double scanned_tuples; /* counts only tuples on scanned pages */ double old_rel_tuples; /* previous value of pg_class.reltuples */ double new_rel_tuples; /* new estimated total # of tuples */ @@ -221,7 +222,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, * We request a full scan if either the table's frozen Xid is now older * than or equal to the requested Xid full-table scan limit; or if the * table's minimum MultiXactId is older than or equal to the requested - * mxid full-table scan limit. + * mxid full-table scan limit. During full scan, we could skip to scan + * pages according to all-frozen bit of visibility map. */ scan_all = TransactionIdPrecedesOrEquals(onerel->rd_rel->relfrozenxid, xidFullScanLimit); @@ -253,7 +255,8 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, * NB: We need to check this before truncating the relation, because that * will change ->rel_pages. */ - if (vacrelstats->scanned_pages < vacrelstats->rel_pages) + if ((vacrelstats->scanned_pages + vacrelstats->vmskipped_frozen_pages) + < vacrelstats->rel_pages) { Assert(!scan_all); scanned_all = false; @@ -274,9 +277,9 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, * Update statistics in pg_class. * * A corner case here is that if we scanned no pages at all because every - * page is all-visible, we should not update relpages/reltuples, because - * we have no new information to contribute. In particular this keeps us - * from replacing relpages=reltuples=0 (which means "unknown tuple + * page is all-visible or all-forzen, we should not update relpages/reltuples, + * because we have no new information to contribute. In particular this keeps + * us from replacing relpages=reltuples=0 (which means "unknown tuple * density") with nonzero relpages and reltuples=0 (which means "zero * tuple density") unless there's some actual evidence for the latter. * @@ -354,10 +357,11 @@ lazy_vacuum_rel(Relation onerel, int options, VacuumParams *params, get_namespace_name(RelationGetNamespace(onerel)), RelationGetRelationName(onerel), vacrelstats->num_index_scans); - appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins\n"), + appendStringInfo(&buf, _("pages: %u removed, %u remain, %u skipped due to pins, %u skipped according to vm\n"), vacrelstats->pages_removed, vacrelstats->rel_pages, - vacrelstats->pinskipped_pages); + vacrelstats->pinskipped_pages, + vacrelstats->vmskipped_frozen_pages); appendStringInfo(&buf, _("tuples: %.0f removed, %.0f remain, %.0f are dead but not yet removable\n"), vacrelstats->tuples_deleted, @@ -480,9 +484,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, * consecutive pages. Since we're reading sequentially, the OS should be * doing readahead for us, so there's no gain in skipping a page now and * then; that's likely to disable readahead and so be counterproductive. - * Also, skipping even a single page means that we can't update - * relfrozenxid, so we only want to do it if we can skip a goodly number - * of pages. + * Also, skipping even a single page according to all-visible bit of + * visibility map means that we might not be able to update relfrozenxid, + * so we on ly want to do if if we can skip a goodly number. On the other hand, + * we count both how many pages we skipped according to all-frozen bit and + * how many pages we froze, so we can update relfrozenxid if the sum of two + * is as many as the number of pages of table. * * Before entering the main loop, establish the invariant that * next_not_all_visible_block is the next block number >= blkno that's not @@ -492,18 +499,18 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, * started skipping blocks, we may as well skip everything up to the next * not-all-visible block. * - * Note: if scan_all is true, we won't actually skip any pages; but we + * Note: if scan_all is true, we might not actually skip any pages; but we * maintain next_not_all_visible_block anyway, so as to set up the * all_visible_according_to_vm flag correctly for each page. * * Note: The value returned by visibilitymap_get_status could be slightly * out-of-date, since we make this test before reading the corresponding * heap page or locking the buffer. This is OK. If we mistakenly think - * that the page is all-visible when in fact the flag's just been cleared, - * we might fail to vacuum the page. But it's OK to skip pages when - * scan_all is not set, so no great harm done; the next vacuum will find - * them. If we make the reverse mistake and vacuum a page unnecessarily, - * it'll just be a no-op. + * that the page is all-visible/all-frozen when in fact the flag's just + * been cleared, we might fail to vacuum the page. But it's OK to skip + * pages when scan_all is not set, so no great harm done; the next vacuum + * will find them. If we make the reverse mistake and vacuum a page + * unnecessarily, it'll just be a no-op. * * We will scan the table's last page, at least to the extent of * determining whether it has tuples or not, even if it should be skipped @@ -536,9 +543,13 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, bool tupgone, hastup; int prev_dead_count; - int nfrozen; + int nfrozen; /* # of tuple is frozen */ + int nalready_frozen; /* # of tuples is already frozen */ + int ntotal_frozen; /* # of frozen tuples in single page */ + int ntup_per_page; Size freespace; bool all_visible_according_to_vm; + bool all_frozen_according_to_vm; bool all_visible; bool all_frozen = true; /* provided all_visible is also true */ bool has_dead_tuples; @@ -570,13 +581,27 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, else skipping_all_visible_blocks = false; all_visible_according_to_vm = false; + all_frozen_according_to_vm = false; } else { - /* Current block is all-visible */ - if (skipping_all_visible_blocks && !scan_all && !FORCE_CHECK_PAGE()) + /* + * This block is at least all-visible according to visibility map. + * Weh check whether this block is all-frozen or not, to skipt to + * scan this page even if scan_all is true. + */ + bool all_frozen = VM_ALL_FROZEN(onerel, blkno, &vmbuffer); + + if (scan_all && all_frozen && !FORCE_CHECK_PAGE()) + { + vacrelstats->vmskipped_frozen_pages++; continue; + } + else if (skipping_all_visible_blocks && !scan_all && !FORCE_CHECK_PAGE()) + continue; + all_visible_according_to_vm = true; + all_frozen_according_to_vm = all_frozen; } vacuum_delay_point(); @@ -792,6 +817,8 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, all_visible = true; has_dead_tuples = false; nfrozen = 0; + nalready_frozen = 0; + ntup_per_page = 0; hastup = false; prev_dead_count = vacrelstats->num_dead_tuples; maxoff = PageGetMaxOffsetNumber(page); @@ -946,6 +973,7 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, else { num_tuples += 1; + ntup_per_page += 1; hastup = true; /* @@ -996,6 +1024,9 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, END_CRIT_SECTION(); } + /* Compute total number of frozen tuples in single page */ + ntotal_frozen = nfrozen + nalready_frozen; + /* * If there are no indexes then we can vacuum the page right now * instead of doing a second scan. @@ -1018,31 +1049,45 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, freespace = PageGetHeapFreeSpace(page); - /* mark page all-visible, if appropriate */ - if (all_visible && !all_visible_according_to_vm) + /* If this page is all visible, consider to set all-visible and all-frozen */ + if (all_visible) { - uint8 flags = VISIBILITYMAP_ALL_VISIBLE; + uint8 flags = 0; + /* mark page all-visible, if appropriate */ + if (!all_visible_according_to_vm) + { + /* + * It should never be the case that the visibility map page is set + * while the page-level bit is clear, but the reverse is allowed + * (if checksums are not enabled). Regardless, set the both bits + * so that we get back in sync. + * + * NB: If the heap page is all-visible but the VM bit is not set, + * we don't need to dirty the heap page. However, if checksums + * are enabled, we do need to make sure that the heap page is + * dirtied before passing it to visibilitymap_set(), because it + * may be logged. Given that this situation should only happen in + * rare cases after a crash, it is not worth optimizing. + */ + PageSetAllVisible(page); + flags |= VISIBILITYMAP_ALL_VISIBLE; + } - if (all_frozen) + /* Mark page as all-frozen, if all tuples are frozen and not marked yet */ + if ((all_frozen || (ntotal_frozen = ntup_per_page)) && + !all_frozen_according_to_vm) + { + PageSetAllFrozen(page); flags |= VISIBILITYMAP_ALL_FROZEN; + } - /* - * It should never be the case that the visibility map page is set - * while the page-level bit is clear, but the reverse is allowed - * (if checksums are not enabled). Regardless, set the both bits - * so that we get back in sync. - * - * NB: If the heap page is all-visible but the VM bit is not set, - * we don't need to dirty the heap page. However, if checksums - * are enabled, we do need to make sure that the heap page is - * dirtied before passing it to visibilitymap_set(), because it - * may be logged. Given that this situation should only happen in - * rare cases after a crash, it is not worth optimizing. - */ - PageSetAllVisible(page); - MarkBufferDirty(buf); - visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr, - vmbuffer, visibility_cutoff_xid, flags); + + if (flags) + { + MarkBufferDirty(buf); + visibilitymap_set(onerel, blkno, buf, InvalidXLogRecPtr, + vmbuffer, visibility_cutoff_xid, flags); + } } /* @@ -1055,7 +1100,12 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, else if (all_visible_according_to_vm && !PageIsAllVisible(page) && VM_ALL_VISIBLE(onerel, blkno, &vmbuffer)) { - elog(WARNING, "page is not marked all-visible but visibility map bit is set in relation \"%s\" page %u", + /* If the all-frozen bit is set then all-visible bit must be set */ + if (all_frozen_according_to_vm) + Assert(VM_ALL_FROZEN(onerel, blkno, &vmbuffer) && + VM_ALL_VISIBLE(onerel, blkno, &vmbuffer)); + + elog(WARNING, "page is not marked all-visible (and all-frozen) but visibility map bit is set in relation \"%s\" page %u", relname, blkno); visibilitymap_clear(onerel, blkno, vmbuffer); } @@ -1063,19 +1113,25 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, /* * It's possible for the value returned by GetOldestXmin() to move * backwards, so it's not wrong for us to see tuples that appear to - * not be visible to everyone yet, while PD_ALL_VISIBLE is already - * set. The real safe xmin value never moves backwards, but + * not be visible to everyone yet, while PD_ALL_VISIBLE (and PD_ALL_FROZEN) + * are already set. The real safe xmin value never moves backwards, but * GetOldestXmin() is conservative and sometimes returns a value * that's unnecessarily small, so if we see that contradiction it just * means that the tuples that we think are not visible to everyone yet - * actually are, and the PD_ALL_VISIBLE flag is correct. + * actually are, and the PD_ALL_VISIBLE (and PD_ALL_FROZEN) flag are + * correct. * * There should never be dead tuples on a page with PD_ALL_VISIBLE * set, however. */ else if (PageIsAllVisible(page) && has_dead_tuples) { - elog(WARNING, "page containing dead tuples is marked as all-visible in relation \"%s\" page %u", + /* If the all-frozen bit is set then all-visible bit must be set */ + if (all_frozen_according_to_vm) + Assert(VM_ALL_FROZEN(onerel, blkno, &vmbuffer) && + VM_ALL_VISIBLE(onerel, blkno, &vmbuffer)); + + elog(WARNING, "page containing dead tuples is marked as all-visible (and all-frozen) in relation \"%s\" page %u", relname, blkno); PageClearAllVisible(page); MarkBufferDirty(buf); @@ -1167,6 +1223,13 @@ lazy_scan_heap(Relation onerel, LVRelStats *vacrelstats, RelationGetRelationName(onerel), tups_vacuumed, vacuumed_pages))); + /* Report how many frozen pages vauum skipped according to visibility map */ + ereport(elevel, + (errmsg_plural("skipped %d frozen page according to visibility map", + "skipped %d frozen pages according to visibility map", + vacrelstats->vmskipped_frozen_pages, + vacrelstats->vmskipped_frozen_pages))); + /* * This is pretty messy, but we split it up so that we can skip emitting * individual parts of the message when not applicable. diff --git a/src/test/regress/expected/visibilitymap.out b/src/test/regress/expected/visibilitymap.out new file mode 100644 index 0000000..767a0ec --- /dev/null +++ b/src/test/regress/expected/visibilitymap.out @@ -0,0 +1,15 @@ +-- +-- Visibility Map +-- +CREATE TABLE vmtest (i INT primary key); +INSERT INTO vmtest SELECT generate_series(1,10000); +VACUUM FREEZE vmtest; +-- Check whether vacuum skips all-frozen pages +\set VERBOSITY terse +VACUUM FREEZE VERBOSE vmtest; +INFO: vacuuming "public.vmtest" +INFO: index "vmtest_pkey" now contains 10000 row versions in 30 pages +INFO: skipped 44 frozen pages according to visibility map +INFO: "vmtest": found 0 removable, 56 nonremovable row versions in 1 out of 45 pages +\set VERBOSITY default +DROP TABLE vmtest; diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index bec0316..9ad2ffc 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -110,3 +110,6 @@ test: event_trigger # run stats by itself because its delay may be insufficient under heavy load test: stats + +# visiblity map and vacuum test cannot concurrently with any test that runs SQL +test: visibilitymap \ No newline at end of file diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 7e9b319..4b4eb07 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -162,3 +162,4 @@ test: with test: xml test: event_trigger test: stats +test: visibilitymap \ No newline at end of file diff --git a/src/test/regress/sql/visibilitymap.sql b/src/test/regress/sql/visibilitymap.sql new file mode 100644 index 0000000..fb9c811 --- /dev/null +++ b/src/test/regress/sql/visibilitymap.sql @@ -0,0 +1,13 @@ +-- +-- Visibility Map +-- +CREATE TABLE vmtest (i INT primary key); +INSERT INTO vmtest SELECT generate_series(1,10000); +VACUUM FREEZE vmtest; + +-- Check whether vacuum skips all-frozen pages +\set VERBOSITY terse +VACUUM FREEZE VERBOSE vmtest; +\set VERBOSITY default + +DROP TABLE vmtest;