This is an automated email from the ASF dual-hosted git repository.
dlmarion pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/accumulo-website.git
The following commit(s) were added to refs/heads/main by this push:
new e7271f402 Updated 4.x design, features, and quickstart user manual
pages (#487)
e7271f402 is described below
commit e7271f402f86aba44aeadb2d4755ce8d90450fdd
Author: Dave Marion <[email protected]>
AuthorDate: Tue Sep 22 07:33:33 2026 -0400
Updated 4.x design, features, and quickstart user manual pages (#487)
---
_docs-4/getting-started/design.md | 87 +++++++++++++++++-----------------
_docs-4/getting-started/features.md | 28 +++++++----
_docs-4/getting-started/quickstart.md | 22 +++------
images/accumulo-monitor-1.png | Bin 215277 -> 92089 bytes
4 files changed, 70 insertions(+), 67 deletions(-)
diff --git a/_docs-4/getting-started/design.md
b/_docs-4/getting-started/design.md
index f16e05c09..eb114bd01 100644
--- a/_docs-4/getting-started/design.md
+++ b/_docs-4/getting-started/design.md
@@ -32,8 +32,7 @@ machines.
## Components
-An instance of Accumulo includes many TabletServers, one Garbage Collector
process,
-one Manager server and many Clients.
+An instance of Accumulo includes one or more Managers, TabletServers,
Compactors, and GarbageCollectors and many clients.
### Tablet Server
@@ -45,7 +44,7 @@ values from all the files it has created and the sorted
in-memory
store.
TabletServers also perform recovery of a tablet
-that was previously on a server that failed, reapplying any writes
+that was previously on a Tablet Server that failed, reapplying any writes
found in the write-ahead log to the tablet.
### Garbage Collector
@@ -57,22 +56,23 @@ They will perform leader election among themselves to
choose a single active ins
### Manager
-The Accumulo Manager is responsible for detecting and responding to
TabletServer
-failure. It tries to balance the load across TabletServer by assigning tablets
carefully
+The Accumulo Manager is responsible for managing Tablet, FaTE transaction, and
external
+compaction state across the cluster. It assigns and balances the Tablets
across TabletServers
and instructing TabletServers to unload tablets when necessary. The Manager
ensures all
tablets are assigned to one TabletServer each, and handles table creation,
alteration,
and deletion requests from clients. The Manager also coordinates startup,
graceful
shutdown and recovery of changes in write-ahead logs when Tablet servers fail.
-Multiple managers may be run. The managers will choose among themselves a
single manager,
-and the others will become backups if the manager should fail.
-
-### Tracer
+The Compaction Coordinator is a function that the primary Manager performs to
coordinate
+the completion of major compactions using Compactor processes. The Coordinator
is responsible
+for identifying what external compaction work needs to be done, and for
communicating with the Compactors
+to assign work, get status updates, and cancel running external compactions.
-The Accumulo Tracer process supports the distributed timing API provided by
Accumulo.
-One to many of these processes can be run on a cluster which will write the
timing
-information to a given Accumulo table for future reference. See the
-[tracing documentation][tracing] for more information.
+Multiple managers may be run concurrently. The managers will choose among
themselves a single manager
+to act as the primary manager and the other managers will act as assistant
managers. Clients will
+connect only to the primary manager. The assistant managers participate in the
execution and
+management of FaTE transactions. The primary manager also participates in the
execution and management
+of FaTE transactions, and also performs tablet management and compaction
coordinator functions.
### Monitor
@@ -88,27 +88,20 @@ Multiple Monitors can be run to provide hot-standby support
in the face of failu
forwarding of logs from remote hosts to the Monitor, only one Monitor process
should be active
at one time. Leader election will be performed internally to choose the active
Monitor.
-### Compactor (experimental)
+### Compactor
-The Accumulo Compactor process is an optional application that can be used to
run compactions
+The Accumulo Compactor process is an application that can be used to run
compactions
outside of the TabletServer. One to many Compactors can be run on a cluster
and each Compactor
process performs one compaction at a time. The Compactor registers its
existence in ZooKeeper
-and communicates with the Compaction Coordinator to retrieve its work and to
register the
-completion status of the compaction. The Compactor process will continue to
perform compactions
-in situations where normal in-TabletServer compactions would fail, such as
TabletServer restart
-and Tablet re-hosting.
+and communicates with the Compaction Coordinator in the primary Manager to
retrieve its work and to register the
+completion status of the compaction.
-### Compaction Coordinator (experimental)
-
-The Accumulo Compaction Coordinator is an optional application that is
required to run compactions
-outside of the TabletServer. The Coordinator is responsible for communicating
with the
-TabletServers, to identify what external compaction work needs to be done, and
the Compactors
-to assign work, get status updates, and cancel running external compactions.
+Compactors also perform the sorting phase of the write-ahead log recovery for
tablets that were
+on a Tablet Server that failed. Once sorting completes, the tablet is assigned
to a Tablet Server,
+which replays the sorted mutations. The Compactor looks for write-ahead log
sorting work before
+requesting the next Major Compaction job.
-Multiple Coordinators may be run. The Coordinators will choose among
themselves a single active Coordinator,
-and the others will become backups if the active Coordinator should fail.
-
-### Scan Server (experimental)
+### Scan Server
The Accumulo Scan Server is an optional application that can be used to run
scans on a tablet's data
outside of the Tablet Server. Many Scan Servers can be run on a cluster and
each Scan Server may run
@@ -120,6 +113,11 @@ Tablet Server. The Scan Server does not have any of the
Tablet data that may res
in-memory maps and the tablet may reference files that have been compacted as
tablet metadata can
be cached within the Scan Server (See Scan Server configuration properties).
+Scan Servers can also perform the sorting phase of the write-ahead log
recovery for tablets that were
+on a Tablet Server that failed. Once sorting completes, the tablet is assigned
to a Tablet Server,
+which replays the sorted mutations. The server property
`sserver.wal.sort.concurrent.max` controls
+the number of threads in the Scan Server that will perform the write-ahead log
sorting.
+
### Client
Accumulo has a client library that can be used to write applications that
write and read
@@ -129,11 +127,11 @@ data to/from Accumulo. See the [Accumulo clients
documentation][clients] for mor
Accumulo stores data in tables, which are partitioned into tablets. Tablets are
partitioned on row boundaries so that all of the columns and values for a
particular
-row are found together within the same tablet. The Manager assigns Tablets to
one
-TabletServer at a time. This enables row-level transactions to take place
without
+row are found together within the same tablet. When required, the Manager
assigns a Tablet to one
+TabletServer. This enables row-level transactions to take place without
using distributed locking or some other complicated synchronization mechanism.
As
clients insert and query data, and as machines are added and removed from the
-cluster, the Manager migrates tablets to ensure they remain available and that
the
+cluster, the Manager may migrate tablets to ensure they remain available and
that the
ingest and query load is balanced across the cluster.
@@ -171,18 +169,19 @@ locality group. The diagram below shows the logical view
and HDFS file view of a
## Compactions
-In order to manage the number of files per tablet, periodically the
TabletServer
-performs Major Compactions of files within a tablet, in which some set of
RFiles
-are combined into one file. The previous files will eventually be removed by
the
-Garbage Collector. This also provides an opportunity to permanently remove
-deleted key-value pairs by omitting key-value pairs suppressed by a delete
entry
-when the new file is created. See the [compaction documentation][compaction]
-for more information.
+In order to manage the number of files per tablet the Manager will periodically
+identify the tablets that need a Major Compaction and will give the highest
+priority job to a Compactor process when it requests the next job. The Major
+Compaction will merge data from an input set of RFiles into a single output
file.
+The previous files will eventually be removed by the Garbage Collector.
+This also provides an opportunity to permanently remove deleted key-value pairs
+by omitting key-value pairs suppressed by a delete entry when the new file is
+created. See the [compaction documentation][compaction] for more information.
## Splitting
-When a table is created it has one tablet. As the table grows its initial
-tablet eventually splits into two tablets. It's likely that one of these
+When a table is created without split points it will have one tablet. As the
table grows its initial
+tablet eventually splits into two tablets. If hosted, it's likely that one of
these
tablets will migrate to another tablet server. As the table continues to grow,
its tablets will continue to split and be migrated. The decision to
automatically split a tablet is based on the size of a tablets files. The
@@ -194,7 +193,8 @@ splitting.
As data is deleted from a table, tablets may shrink. Over time this can lead
to small or empty tablets. To deal with this, the [merging of tablets][merging]
-was introduced in Accumulo 1.4.
+was introduced in Accumulo 1.4. Accumulo 4.0 introduces [automatic
merging][merging2]
+of tablets to optionally automate merging of tablets based on configurable
thresholds.
## Fault-Tolerance
@@ -209,7 +209,7 @@ grouped by tablet. TabletServers can quickly apply the
mutations from the sorte
that are destined for the tablets they have now been assigned.
TabletServer failures are noted on the Manager's monitor page, accessible via
-`http://manager-address:9995/monitor`.
+`http://monitor-address:9995`.
@@ -218,5 +218,6 @@ TabletServer failures are noted on the Manager's monitor
page, accessible via
[tracing]: {% durl troubleshooting/tracing %}
[clients]: {% durl getting-started/clients %}
[merging]: {% durl getting-started/table_configuration#merging-tablets %}
+[merging2]: {% durl administration/merging %}
[compaction]: {% durl getting-started/table_configuration#compaction %}
[caching]: {% durl administration/caching %}
diff --git a/_docs-4/getting-started/features.md
b/_docs-4/getting-started/features.md
index 2d8107652..68ca28493 100644
--- a/_docs-4/getting-started/features.md
+++ b/_docs-4/getting-started/features.md
@@ -64,9 +64,9 @@ can share a pool of datanodes.
### Manager fail over
Multiple managers can be configured. Zookeeper locks are used to determine
-which manager is active. The remaining managers simply wait for the current
-manager to lose its lock. Current manager state is held in the metadata table
-and Zookeeper.
+which manager is the primary manager. The remaining managers will attempt to
+take on the primary manager role if the current primary manager loses its lock.
+Current manager state is held in the metadata table and Zookeeper.
### Logical time
@@ -305,14 +305,18 @@ Accumulo provides two BlockCacheManager implementations
(LruBlockCacheManager an
TinyLfuBlockCacheManager) that construct on-heap block caches. Users can
provide
alternate BlockCacheManager implementations using the property
`tserver.cache.manager.class`.
+### On-Demand Tablet Unloader
+
+Tablets that have their availability set to on-demand can be unhosted.
Accumulo has a default
+implementation that unloads idle tablets after a configurable amount of time.
+
## General Administration
### Monitor page
The [Accumulo Monitor][monitor] provides basic information about the system
health and
-performance. It displays table sizes, ingest and query statistics, server
-load, and last-update information. It also allows the user to view recent
-diagnostic logs and traces.
+performance. It displays information about the server processes, table sizes,
ingest,
+query and compaction statistics, FaTE and recovery information, and
last-update information.
<a class="p-3 border rounded d-block" href="/images/accumulo-monitor-1.png">
<img src="/images/accumulo-monitor-1.png" class="img-fluid rounded"
alt="monitor overview"/>
@@ -386,9 +390,8 @@ used to accommodate new data patterns in an existing table.
### Tablet Merging
-Tablet merging is a new feature. Merging of tablets can be requested in the
-shell; Accumulo does not merge tablets automatically. If desired, the METADATA
-tablets can be merged.
+Merging of tablets can be requested in the shell or tables can be configured to
+merge tablets automatically.
### Table Cloning
@@ -413,6 +416,13 @@ Added an operation to efficiently delete a range of rows
from a table. Tablets
that fall completely within a range are simply dropped. Tablets overlapping the
beginning and end of the range are split, compacted, and then merged.
+### Tablet Hosting
+
+User table tablets are hosted optionally be default. Users can configure
tablets
+for their tables to always or never be hosted. Optionally hosted tablets will
be
+assigned to a Tablet Server if needed to support a user operation and then
unhosted
+after a configurable period of time of inactivity.
+
[FATE]: {% durl administration/fate %}
[maven-accumulo-plugin]: {{ site.baseurl
}}/release/accumulo-1.6.0/#maven-plugin
[ConditionalMutation]: {% jurl
org.apache.accumulo.core.data.ConditionalMutation %}
diff --git a/_docs-4/getting-started/quickstart.md
b/_docs-4/getting-started/quickstart.md
index ba4bce100..2b9485e56 100644
--- a/_docs-4/getting-started/quickstart.md
+++ b/_docs-4/getting-started/quickstart.md
@@ -25,11 +25,7 @@ process name referenced in this documentation.
## Setup for testing or development
If you are setting up Accumulo for **testing or development,** consider using
-the following tools:
-
-* [Uno] sets up Accumulo on a single machine for development
-* [Muchos] sets up Accumulo on a cluster (optionally launched in Amazon EC2 and
- Microsoft Azure VM)
+[Uno] to set up a single instance Accumulo cluster.
If you are setting up Accumulo for a **production** environment, follow the
instructions below.
@@ -138,7 +134,7 @@ instance.
The following command will perform the initialization.
```
-accumulo init
+accumulo inst init
```
The initialization command will prompt for the following information.
@@ -176,7 +172,7 @@ command followed by the service name. For example, to start
only the tserver,
run:
```
-accumulo tserver
+accumulo proc tserver
```
The process will run in the foreground. Use ctrl-c to quit.
@@ -226,11 +222,9 @@ file contains sections that should be configured with a
list of node names in ya
* [monitor] : Node where Accumulo monitoring web server is run.
* [tserver] : Accumulo worker processes. List all of the nodes where tablet
servers should run.
- * [sserver] : Optional. List of all nodes where scan servers should run.
- * [compaction.coordinator] : Optional. Must specify one node. Can specify a
few
- for fault tolerance.
- * [compaction.compactor] : Optional. Accumulo external compactor processes.
List of
+ * [compactor] : Accumulo external compactor processes. List of
all nodes where compactors should run.
+ * [sserver] : Optional. List of all nodes where scan servers should run.
The Accumulo, Hadoop, and Zookeeper software should be present at the same
location on every node. Also, the files in the `conf` directory must be copied
to
@@ -290,10 +284,8 @@ When finished, use the following commands to stop Accumulo:
[manager]: {% durl configuration/files#managers %}
[tserver]: {% durl configuration/files#tservers %}
[tracer]: {% durl configuration/files#tracers %}
-[compaction.coordinator]: {% durl configuration/files#compaction%20coordinator
%}
-[compaction.compactor]: {% durl configuration/files#compaction%20compactor %}
+[compactor]: {% durl configuration/files#compaction%20compactor %}
[sserver]: {% durl configuration/files#sserver %}
-[Uno]: https://github.com/apache/fluo-uno
-[Muchos]: https://github.com/apache/fluo-muchos
+[Uno]: https://github.com/apache/accumulo-uno
[Erasure Coding]:
https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html
[ec-guide]: {% durl administration/erasure-coding %}
diff --git a/images/accumulo-monitor-1.png b/images/accumulo-monitor-1.png
index c6a711a49..6a83f3f25 100644
Binary files a/images/accumulo-monitor-1.png and
b/images/accumulo-monitor-1.png differ
