From fd18641adfe19ecf12a4f6a17a07bd5ffb272d48 Mon Sep 17 00:00:00 2001 From: Kommi Date: Mon, 11 Mar 2019 15:44:44 +1100 Subject: [PATCH 3/3] Table access method API explanation All the table access method API's and their details are explained. --- doc/src/sgml/am.sgml | 794 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 789 insertions(+), 5 deletions(-) diff --git a/doc/src/sgml/am.sgml b/doc/src/sgml/am.sgml index b2a97f20aa..455e7a10c6 100644 --- a/doc/src/sgml/am.sgml +++ b/doc/src/sgml/am.sgml @@ -18,14 +18,798 @@ All Tables in PostgreSQL are the primary data store. Each table is stored as its own physical relation - and so is described by an entry in the pg_class - catalog. The table contents are entirely under the control of its - access method. (All the access methods furthermore use the standard page - layout described in .) + and is described by an entry in the pg_class + catalog. A table's content is entirely controlled by its access method, although + all access methods use the same standard page layout described in . - + + Table access method API + + + Each table access method is described by a row in the + pg_am system + catalog. The pg_am entry specifies a type + of the access method and a handler function for the + access method. These entries can be created and deleted using the + and SQL commands. + + + + A table access method handler function must be declared to accept a + single argument of type internal and to return the + pseudo-type table_am_handler. The argument is a dummy value that + simply serves to prevent handler functions from being called directly from + SQL commands. The result of the function must be a palloc'd struct of + type TableAmRoutine, which contains everything + that the core code needs to know to make use of the table access method. + The TableAmRoutine struct, also called the access + method's API struct, includes fields specifying assorted + fixed properties of the access method, such as whether it can support + bitmap scans. More importantly, it contains pointers to support + functions for the access method, which do all of the real work to access + tables. These support functions are plain C functions and are not + visible or callable at the SQL level. The support functions are described + in TableAmRoutine structure. For more details, please + refer the file src/include/access/tableam.h. + + + + Any new TABLE ACCSESS METHOD developers can refer the exisitng HEAP + implementation present in the src/backend/heap/heapam_handler.c for more details of + how it is implemented for HEAP access method. + + + + There are different type of API's that are defined and those details are below. + + + + Slot implementation functions + + + +const TupleTableSlotOps *(*slot_callbacks) (Relation rel); + + + This API expects the function should return the slot implementation that is specific to the AM. + Following are the predefined types of slot implementations that are available, + TTSOpsVirtual, TTSOpsHeapTuple, + TTSOpsMinimalTuple and TTSOpsBufferHeapTuple. + The AM implementations can use any one of them. For more details of these slot + specific implementations, you can refer src/include/executor/tuptable.h. + + + + + Table scan functions + + + The following API's are used for scanning of a table. + + + + +TableScanDesc (*scan_begin) (Relation rel, + Snapshot snapshot, + int nkeys, struct ScanKeyData *key, + ParallelTableScanDesc pscan, + bool allow_strat, + bool allow_sync, + bool allow_pagemode, + bool is_bitmapscan, + bool is_samplescan, + bool temp_snap); + + + This API to start a scan of a relation pointed by rel and returns the + TableScanDesc, which will be typically embed in a larger AM specific, + strcut. + + The nkeys indicates results needs to be filtered based on the key. + The pscan can be used by the AM, in case if it supports parallel scan. + The parameters allow_strat, allow_sync and allow_pagemode + are used for specifying whether the scan strategy, as whether it supports synchronize scans or + pagemode scans (although every AM is not required to support these). + + The parameters is_bitmapscan and is_samplescan are used to + specify whether the scan is intended to support those type of scans are not? + + The temp_snap indicates the provided snapshot is a temporary allocated and + it needs to be freed at the scan end. + + + + +void (*scan_end) (TableScanDesc scan); + + + This API to end the scan that is started by the API scan_begin + by releasing the resources. TableScanDesc.rs_snapshot + needs to be unregistered and it can be deallocated based on TableScanDesc.temp_snap. + + + + +void (*scan_rescan) (TableScanDesc scan, struct ScanKeyData *key, bool set_params, + bool allow_strat, bool allow_sync, bool allow_pagemode); + + + This API to restart the given relation scan that is already started by the + API scan_begin. if set_params is set + to true, consider the provided options into the scan. + + + + +TupleTableSlot *(*scan_getnextslot) (TableScanDesc scan, + ScanDirection direction, TupleTableSlot *slot); + + + This API to return the next satisified tuple from the scan started by the API + scan_begin and store it in the slot. + + + + + + parallel table scan functions + + + The following API's are used to perform the parallel table scan. + + + + +Size (*parallelscan_estimate) (Relation rel); + + + This API to return the total size that is required for the AM to perform + the parallel table scan. The requied size must include the ParallelTableScanDesc + which is typically embed in the AM specific struct. + + + + +Size (*parallelscan_initialize) (Relation rel, ParallelTableScanDesc pscan); + + + This API to perform the initialization of the pscan + that is required for the parallel scan to be performed by the AM and also return + the size that is estimated by the parallelscan_estimate. + + + + +void (*parallelscan_reinitialize) (Relation rel, ParallelTableScanDesc pscan); + + + This API to reinitalize the parallel scan structure pointed by the pscan + for the same relation. + + + + + + Index scan functions + + + +struct IndexFetchTableData *(*index_fetch_begin) (Relation rel); + + + This API to prepare fetching tuples from the relation, as needed when fetching + from index scan. The API needs to return the allocated and initialized IndexFetchTableData + strutucture, which is typically embed in the AM specific struct. + + + + +void (*index_fetch_reset) (struct IndexFetchTableData *data); + + + This API to reset the index fetch, typically it releases the AM specific resources + that are held by IndexFetchTableData of a index scan. + + + + +void (*index_fetch_end) (struct IndexFetchTableData *data); + + + This API to release AM-specific resources held by the IndexFetchTableData + and free the memory of IndexFetchTableData itself. + + + + +bool (*index_fetch_tuple) (struct IndexFetchTableData *scan, + ItemPointer tid, + Snapshot snapshot, + TupleTableSlot *slot, + bool *call_again, bool *all_dead); + + + This API to fetch the tuple pointed by tid of a relation and store it in the + slot after performing visibility check according the provided snapshot. + Returns true when the tuple is found or false. + + The call_again is false when the API is called for the first time with the tid, + in case if there are any potential match for another tuple, call_again must be + set to true to indicate the caller to execute the API again to fetch the tuple. + + The all_dead is not NULL, should be set to true if by the API function iff it + is guaranteed that no backend needs to see that tuple. Index AMs can use that do avoid returning + that tid in future searches. + + + + + + Non modifying tuple functions + + + +bool (*tuple_fetch_row_version) (Relation rel, + ItemPointer tid, + Snapshot snapshot, + TupleTableSlot *slot); + + + This API to fetches the latest tuple specified by the ItemPointer tid + and store it in the slot after doing a visibilty test according to the + snapshot. If a tuple was found and passed visibility test return true, + otherwise false. + + For e.g, in the case if Heap AM, the update chains are created whenever + the tuple is updated, so the function should fetch the latest tuple. + + + + +void (*tuple_get_latest_tid) (Relation rel, + Snapshot snapshot, + ItemPointer tid); + + + This API to get the latest version of the tuple based on the specified ItemPointer tid. + For e.g, in the case of Heap AM, the update chains are created whenever any tuple is updated. + This API is used to find out latest ItemPointer. + + + + +bool (*tuple_satisfies_snapshot) (Relation rel, + TupleTableSlot *slot, + Snapshot snapshot); + + + This API performs the tuple visibility that is present in the slot + based on provided snapshot and returns true if the current tuple is visible, + otherwise false. Some AMs might modify the data underlying the tuple as a side-effect. + If so they ought to mark the relevant buffer dirty. + + + + +bool (*tuple_fetch_follow) (struct IndexFetchTableData *scan, + ItemPointer tid, + Snapshot snapshot, + TupleTableSlot *slot, + bool *call_again, bool *all_dead); + + + This API is used to fetch the tuple pointed by the ItemPointer based on the + IndexFetchTableData and store it in the specified slot and also updates the flags. + This API is called from the index scan operation. + + + + +TransactionId (*compute_xid_horizon_for_tuples) (Relation rel, + ItemPointerData *items, + int nitems); + + + This API to get the newest xid among the provided tuples by items. This is used + to compute what snapshots to conflict with the items when replaying WAL records + for page-level index vacuums. + + + + + + Manipulation of physical tuples functions + + + +void (*tuple_insert) (Relation rel, TupleTableSlot *slot, CommandId cid, + int options, struct BulkInsertStateData *bistate); + + + This API to insert a tuple from a slot into AM routine. + + The cid is command identifier used to specify the number + of the comamnd that is getting processed. + + The options bitmask allows to specify options that allow + to change the behaviour of the AM. Several options might be ignored by AMs + not supporting them. + If the TABLE_INSERT_SKIP_WAL option is specified, the new + tuple will not necessarily logged to WAL, even for a non-temp relation. It is + the AMs choice whether this optimization is supported. + If the TABLE_INSERT_SKIP_FSM option is specified, AMs are + free to not reuse free space in the relation. This can save some cycles when + we know the relation is new and doesn't contain useful amounts of free space. + It's commonly passed directly to RelationGetBufferForTuple, see for more info. + If the TABLE_INSERT_FROZEN option can only be specified for + inserts into relfilenodes created during the current subtransaction and when + there are no prior snapshots or pre-existing portals open. This causes rows to + be frozen, which is an MVCC violation and requires explicit options chosen by user. + If the TABLE_INSERT_NO_LOGICAL can only be specified to + indicate the AM to force-disables the emitting of logical decoding information + for the tuple. This should solely be used during table rewrites where + RelationIsLogicallyLogged(relation) is not yet accurate for the new relation. + + The BulkInsertState object (if any; bistate can be NULL for default + behavior) is also just passed through to RelationGetBufferForTuple. + + On return the slot's tts_tid and tts_tableOid are updated to reflect the + insertion. + + + + +void (*tuple_insert_speculative) (Relation rel, + TupleTableSlot *slot, + CommandId cid, + int options, + struct BulkInsertStateData *bistate, + uint32 specToken); + + + This API is similar like tuple_insert API, but it perform a + "speculative insertion". This API is used to backed out afterwards without aborting + the whole transaction. + + Other sessions can wait for the speculative insertion to be confirmed, turning it + into a regular tuple, or aborted, as if it never existed. Speculatively inserted + tuples behave as "value locks" of short duration, used to implement + INSERT .. ON CONFLICT. + + A transaction having performed a speculative insertion has to either abort, or finish + the speculative insertion with table_complete_speculative(). + + + + +void (*tuple_complete_speculative) (Relation rel, + TupleTableSlot *slot, + uint32 specToken, + bool succeeded); + + + This API to complete the speculative insertion of a tuple started in the same transaction + by tuple_insert_speculative, It is invoked after finishing the index insert. + If succeeded is true, the tuple is fully inserted, if false it should b + removed. + + + + +TM_Result (*tuple_delete) (Relation rel, + ItemPointer tid, + CommandId cid, + Snapshot snapshot, + Snapshot crosscheck, + bool wait, + TM_FailureData *tmfd, + bool changingPart); + + + This API to delete a tuple of the relation pointed by the ItemPointer tid + and returns the result of the operation. + + The cid is a command identifier, used for the visibility test to identify + the tuple according to the snapshot snapshot. + The crosscheck is not null, use it for verifying it against the visibilty + test. + The wait is true indicates the process to wait for any conflicting transactions + to either commit/rollback. + + The following two parameters must be outputed by the API function. + + The tmfd should be set with proper details when the tuple delete operation fails. + The data that needs to fill in case failure, refer TM_FailureData. + The changingPart true iff the tuple is being moved to another partition + table due to an update of the partition key. Otherwise, false. + + + + +TM_Result (*tuple_update) (Relation rel, + ItemPointer otid, + TupleTableSlot *slot, + CommandId cid, + Snapshot snapshot, + Snapshot crosscheck, + bool wait, + TM_FailureData *tmfd, + LockTupleMode *lockmode, + bool *update_indexes); + + + This API to updates a tuple of the relation pointed by the ItemPointer otid + with the new tuple from slot and returns the result of the operation. + + The cid is a command identifier, used for the visibility test to identify + the tuple according to the snapshot snapshot. + The crosscheck is not null, use it for verifying it against the visibilty + test. + The wait is true indicates the process to wait for any conflicting transactions + to either commit/rollback. + + The following three parameters must be outputed by the API function. + + The tmfd should be set with proper details when the tuple update operation fails. + The data that needs to fill in case failure, refer TM_FailureData. + The lockmode filled with lock mode acquired on tuple. + The update_indexes true if new index entries are required for this tuple. + Otherwise false. + + On return the slot's tts_tid and tts_tableOid are updated to reflect the update. In particular, + slot->tts_tid is set to the TID where the new tuple was inserted, and its HEAP_ONLY_TUPLE flag + is set iff a HOT update was done. + + + + +void (*multi_insert) (Relation rel, TupleTableSlot **slots, int nslots, + CommandId cid, int options, struct BulkInsertStateData *bistate); + + + This API to perform insertion of multiple tuples from slots into the relation + for faster data insertion. Refer tuple_insert for more details about parameters and + return values. + + + + +TM_Result (*tuple_lock) (Relation rel, + ItemPointer tid, + Snapshot snapshot, + TupleTableSlot *slot, + CommandId cid, + LockTupleMode mode, + LockWaitPolicy wait_policy, + uint8 flags, + TM_FailureData *tmfd); + + + This API to lock a tuple pointed by the ItemPointer tid with the specified mode + and return the result of the operation. + + The cid is a command identifier, used for the visibility test to identify + the tuple according to the snapshot snapshot. + The mode lock mode desired. + The wait_policy indicates the operation in case if the tuple lock is not available. + The flags allows to specify options such as TUPLE_LOCK_FLAG_LOCK_UPDATE_IN_PROGRESS + to follow the update chain to also lock descendant tuples if lock modes don't conflict. or + TUPLE_LOCK_FLAG_FIND_LAST_VERSION update chain and lock latest version. + + The following two parameters must be outputed by the API function. + + The tmfd should be set with proper details when the tuple update operation fails. + The slot contains the locked target tuple. + + + + +void (*finish_bulk_insert) (Relation rel, int options); + + + This API to perform the operations necessary to complete insertions made + via tuple_insert and multi_insert with a + BulkInsertState specified. This e.g. may e.g. used to flush the relation when + inserting with skipping WAL or may be no operation. + + + + + + DDL related functions + + + +void (*relation_set_new_filenode) (Relation rel, + char persistence, + TransactionId *freezeXid, + MultiXactId *minmulti); + + + This API to create the storage file for the relation rel, with presistence set to + persistence that is necessary to store the tuples of the relation. + freezeXid, minmulti are set to the xid / multixact + horizon for the table that pg_class.{relfrozenxid, relminmxid} have to be set to. For e.g, the Heap AM, + should create the relfilenode that is necessary to store the heap tuples. + + + + +void (*relation_nontransactional_truncate) (Relation rel); + + + This API is used to remove all table contents from `rel`, in a non-transactional manner. + Non-transactional meaning that there's no need to support rollbacks. This commonly only + is used to perform truncations for relfilenodes created in the current transaction. + This operation is not non-reversible. + + + + +void (*relation_copy_data) (Relation rel, RelFileNode newrnode); + + + This API to perform the copy of the relation rel from existing filenode + to the new filenode newrnode and removes the existing filenode. + + + + +void (*relation_copy_for_cluster) (Relation NewHeap, Relation OldHeap, Relation OldIndex, + bool use_sort, + TransactionId OldestXmin, TransactionId FreezeXid, MultiXactId MultiXactCutoff, + double *num_tuples, double *tups_vacuumed, double *tups_recently_dead); + + + This API to make a copy data from OldHeap into NewHeap, + as part of a CLUSTER or VACUUM FULL. + + If use_sort is true, the table contents are sorted appropriate for + OldIndex; if use_sort is false and OldIndex + is not InvalidOid, the data is copied in that index's order; if use_sort + is false and OidIndex is InvalidOid, no sorting is performed. + + OldestXmin, FreezeXid, MultiXactCutoff + can be used to clean the dead tuples of the table. + + On successfule operation, num_tuples, tups_vacuumed, tups_recently_dead + will contain statistics computed while copying for the relation. Not all might make sense for every AM. + + + + +void (*relation_vacuum) (Relation onerel, int options, + struct VacuumParams *params, BufferAccessStrategy bstrategy); + + + This API performs vacuuming of the relation based on the specified params. + It Gathers all the dead tuples of the relation and clean them including + the indexes. + + This API Perform VACUUM on the relation. The VACUUM + can be user triggered or by autovacuum. The specific actions performed + by the AM will depend heavily on the individual AM. For eg- heapAM, It Gathers all the + dead tuples of the relation and clean them including the indexes + + Note that neither VACUUM FULL (and CLUSTER), nor + ANALYZE go through this routine, even if (in the latter case), part of + the same VACUUM command. + + + + +void (*scan_analyze_next_block) (TableScanDesc scan, BlockNumber blockno, + BufferAccessStrategy bstrategy); + + + This API prepares the block blockno to analyze of table scan scan. + The scan needs to have been started with table_beginscan_analyze(). + Note that this routine might acquire resources like locks that are held until + table_scan_analyze_next_tuple() returns false. + + Returns false if block is unsuitable for sampling, true otherwise. + + + + +bool (*scan_analyze_next_tuple) (TableScanDesc scan, TransactionId OldestXmin, + double *liverows, double *deadrows, TupleTableSlot *slot); + + + This API iterate over tuples in the block selected with table_scan_analyze_next_block() + If a tuple that's suitable for sampling is found, returns true and a tuple is stored in slot. + Otherwise returns false. + + The liverows and deadrows are incremented according to the encountered + tuples. + + + + +double (*index_build_range_scan) (Relation heap_rel, + Relation index_rel, + IndexInfo *index_nfo, + bool allow_sync, + bool anyvisible, + BlockNumber start_blockno, + BlockNumber end_blockno, + IndexBuildCallback callback, + void *callback_state, + TableScanDesc scan); + + + This API to scan the table to find the tuples to be indexed from the specified + blocks of a given relation and insert them into the specified index using the + provided the callback function. + + This is called back from an access-method-specific index build procedure + after the AM has done whatever setup it needs. The parent heap relation + heap_rel is scanned to find tuples that should be entered + into the index index_rel. Each such tuple is passed to + the AM's callback routine callback, which does the right + things to add it to the new index. After we return, the AM's index build + procedure does whatever cleanup it needs. + + The callback_state is member needs to be passed to the + callback when it is invoked from the AM specific function. + + The allow_sync specifies the scan on the relation should follow + synchronize_seqscans configuration parameter. + + The index_info can be used to pass back some infromation + related to the AM. For eg- in heapAM, indexInfo->ii_BrokenHotChain + to true if we detect any potentially broken HOT chains. Currently, we set + this if there are any RECENTLY_DEAD or DELETE_IN_PROGRESS entries in a HOT chain, + without trying very hard to detect whether they're really incompatible with + the chain tip. This need to be generalized for other AMs later. + + When anyvisible mode is requested, all tuples visible to + any transaction are indexed and counted as live, including those inserted + or deleted by transactions that are still in progress. + + The start_blockno and end_blockno are + used to specify the range of the blocks that needs to be scanned on the relation. + + Upon successful execution, The total count of live tuples is returned. This + is for updating pg_class statistics. + + + + +void (*index_validate_scan) (Relation heap_rel, + Relation index_rel, + IndexInfo *index_info, + Snapshot snapshot, + struct ValidateIndexState *state); + + + This API to perform second table scan in a concurrent index build. + + The table heap_rel scanned to find the tuples and insert + them into index_rel according to the given snapshot snapshot + by verifying their ItemPointerData in the provided ValidateIndexState struct; + this API is used as the last phase of a concurrent index build. + + + + + + planner functions + + + +void (*relation_estimate_size) (Relation rel, int32 *attr_widths, + BlockNumber *pages, double *tuples, double *allvisfrac); + + + This API estimates the current size of the relation rel and also + returns the number of pages, tuples corresponding relation. + + If attr_widths isn't NULL, It may contains the previously cached relation + attribute widhts, AM must fill this in if we have need to compute the attribute widths for + estimation purposes. + + It also returns allvisfrac the fraction of the pages that are marked + all-visible in the visibilty map. + + + + + + executor functions + + + +bool (*scan_bitmap_next_block) (TableScanDesc scan, + TBMIterateResult *tbmres); + + + This API prepare to fetch / check / return tuples from tbmres + structure blockno as part of a bitmap table scan. + scan that was started via table_beginscan_bm(). + Return false if there's no tuples to be found on the page, true otherwise. + + If tbmres->blockno is -1, this is a lossy scan + and all visible tuples on the page have to be returned, otherwise the tuples at + offsets in tbmres->offsets need to be returned. + + This is an optional callback, but either both scan_bitmap_next_block + and scan_bitmap_next_tuple need to exist, or neither. + + + + +bool (*scan_bitmap_next_tuple) (TableScanDesc scan, + TupleTableSlot *slot); + + + This API to get the next tuple from the set of tuples of a given page specified in the scan descriptor + and return the provided slot; returns false in case if there are no more tuples. + + This API fetch the next tuple of a bitmap table scan into slot + and return true if a visible tuple was found, false otherwise. + + For some AMs it will make more sense to do all the work referencing tbmres + contents in scan_bitmap_next_block, for others it might be + better to defer more work to this callback. + + This is an optional callback, but either both scan_bitmap_next_block + and scan_bitmap_next_tuple need to exist, or neither. + + + + +bool (*scan_sample_next_block) (TableScanDesc scan, + struct SampleScanState *scanstate); + + + This API to prepare to fetch tuples from the next block in a sample scan. + Return false if the sample scan is finished, true otherwise. scan + was started via table_beginscan_sampling(). + + Typically this will first determine the target block by call the + scanstate->tsmroutine->NextSampleBlock if not NULL, + or alternatively perform a sequential scan over all blocks. The determined + block is then typically read and pinned, + + As the TsmRoutine interface is block based, the blocks needs to be passed + to NextSampleBlock() to return the sampled block. + + Note that it's not acceptable to hold deadlock prone resources such as lwlocks + until scan_sample_next_tuple() has exhausted the tuples on the + block - the tuple is likely to be returned to an upper query node, and the + next call could be off a long while. Holding buffer pins etc is obviously OK. + + Currently it is required to implement this interface, as there's no + alternative way (contrary e.g. to bitmap scans) to implement sample + scans. If infeasible to implement the AM may raise an error. + + + + +bool (*scan_sample_next_tuple) (TableScanDesc scan, + struct SampleScanState *scanstate, + TupleTableSlot *slot); + + + This API must determine the next tuple and store it in slot + from the selected block using the TsmRoutine's NextSampleTuple() + callback. + + This API needs to perform visibilty checks according to snapshot and return only + the valid tuple. + + The TsmRoutine interface assumes that there's a maximum offset + on a given page, so if that doesn't apply to an AM, it needs to emulate that + assumption somehow. + + + + + + Overview of Index access methods -- 2.20.1.windows.1