http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/getting-started/design.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/getting-started/design.md b/_docs-unreleased/getting-started/design.md new file mode 100644 index 0000000..251ae9f --- /dev/null +++ b/_docs-unreleased/getting-started/design.md @@ -0,0 +1,165 @@ +--- +title: Accumulo Design +category: getting-started +order: 1 +--- + +## Data Model + +Accumulo provides a richer data model than simple key-value stores, but is not a +fully relational database. Data is represented as key-value pairs, where the key and +value are comprised of the following elements: + + + +All elements of the Key and the Value are represented as byte arrays except for +Timestamp, which is a Long. Accumulo sorts keys by element and lexicographically +in ascending order. Timestamps are sorted in descending order so that later +versions of the same Key appear first in a sequential scan. Tables consist of a set of +sorted key-value pairs. + +## Architecture + +Accumulo is a distributed data storage and retrieval system and as such consists of +several architectural components, some of which run on many individual servers. +Much of the work Accumulo does involves maintaining certain properties of the +data, such as organization, availability, and integrity, across many commodity-class +machines. + +## Components + +An instance of Accumulo includes many TabletServers, one Garbage Collector process, +one Master server and many Clients. + +### Tablet Server + +The TabletServer manages some subset of all the tablets (partitions of tables). This includes receiving writes from clients, persisting writes to a +write-ahead log, sorting new key-value pairs in memory, periodically +flushing sorted key-value pairs to new files in HDFS, and responding +to reads from clients, forming a merge-sorted view of all keys and +values from all the files it has created and the sorted in-memory +store. + +TabletServers also perform recovery of a tablet +that was previously on a server that failed, reapplying any writes +found in the write-ahead log to the tablet. + +### Garbage Collector + +Accumulo processes will share files stored in HDFS. Periodically, the Garbage +Collector will identify files that are no longer needed by any process, and +delete them. Multiple garbage collectors can be run to provide hot-standby support. +They will perform leader election among themselves to choose a single active instance. + +### Master + +The Accumulo Master is responsible for detecting and responding to TabletServer +failure. It tries to balance the load across TabletServer by assigning tablets carefully +and instructing TabletServers to unload tablets when necessary. The Master ensures all +tablets are assigned to one TabletServer each, and handles table creation, alteration, +and deletion requests from clients. The Master also coordinates startup, graceful +shutdown and recovery of changes in write-ahead logs when Tablet servers fail. + +Multiple masters may be run. The masters will choose among themselves a single master, +and the others will become backups if the master should fail. + +### Tracer + +The Accumulo Tracer process supports the distributed timing API provided by Accumulo. +One to many of these processes can be run on a cluster which will write the timing +information to a given Accumulo table for future reference. Seeing the section on +Tracing for more information on this support. + +### Monitor + +The Accumulo Monitor is a web application that provides a wealth of information about +the state of an instance. The Monitor shows graphs and tables which contain information +about read/write rates, cache hit/miss rates, and Accumulo table information such as scan +rate and active/queued compactions. Additionally, the Monitor should always be the first +point of entry when attempting to debug an Accumulo problem as it will show high-level problems +in addition to aggregated errors from all nodes in the cluster. See the [Accumulo monitor documentation][monitor] +for more information. + +Multiple Monitors can be run to provide hot-standby support in the face of failure. Due to the +forwarding of logs from remote hosts to the Monitor, only one Monitor process should be active +at one time. Leader election will be performed internally to choose the active Monitor. + +### Client + +Accumulo includes a client library that is linked to every application. The client +library contains logic for finding servers managing a particular tablet, and +communicating with TabletServers to write and retrieve key-value pairs. + +## Data Management + +Accumulo stores data in tables, which are partitioned into tablets. Tablets are +partitioned on row boundaries so that all of the columns and values for a particular +row are found together within the same tablet. The Master assigns Tablets to one +TabletServer at a time. This enables row-level transactions to take place without +using distributed locking or some other complicated synchronization mechanism. As +clients insert and query data, and as machines are added and removed from the +cluster, the Master migrates tablets to ensure they remain available and that the +ingest and query load is balanced across the cluster. + + + +## Tablet Service + +When a write arrives at a TabletServer it is written to a Write-Ahead Log and +then inserted into a sorted data structure in memory called a MemTable. When the +MemTable reaches a certain size, the TabletServer writes out the sorted +key-value pairs to a file in HDFS called a Relative Key File (RFile), which is a +kind of Indexed Sequential Access Method (ISAM) file. This process is called a +minor compaction. A new MemTable is then created and the fact of the compaction +is recorded in the Write-Ahead Log. + +When a request to read data arrives at a TabletServer, the TabletServer does a +binary search across the MemTable as well as the in-memory indexes associated +with each RFile to find the relevant values. If clients are performing a scan, +several key-value pairs are returned to the client in order from the MemTable +and the set of RFiles by performing a merge-sort as they are read. + +## Compactions + +In order to manage the number of files per tablet, periodically the TabletServer +performs Major Compactions of files within a tablet, in which some set of RFiles +are combined into one file. The previous files will eventually be removed by the +Garbage Collector. This also provides an opportunity to permanently remove +deleted key-value pairs by omitting key-value pairs suppressed by a delete entry +when the new file is created. + +## Splitting + +When a table is created it has one tablet. As the table grows its initial +tablet eventually splits into two tablets. Its likely that one of these +tablets will migrate to another tablet server. As the table continues to grow, +its tablets will continue to split and be migrated. The decision to +automatically split a tablet is based on the size of a tablets files. The +size threshold at which a tablet splits is configurable per table. In addition +to automatic splitting, a user can manually add split points to a table to +create new tablets. Manually splitting a new table can parallelize reads and +writes giving better initial performance without waiting for automatic +splitting. + +As data is deleted from a table, tablets may shrink. Over time this can lead +to small or empty tablets. To deal with this, merging of tablets was +introduced in Accumulo 1.4. This is discussed in more detail later. + +## Fault-Tolerance + +If a TabletServer fails, the Master detects it and automatically reassigns the tablets +assigned from the failed server to other servers. Any key-value pairs that were in +memory at the time the TabletServer fails are automatically reapplied from the Write-Ahead +Log(WAL) to prevent any loss of data. + +Tablet servers write their WALs directly to HDFS so the logs are available to all tablet +servers for recovery. To make the recovery process efficient, the updates within a log are +grouped by tablet. TabletServers can quickly apply the mutations from the sorted logs +that are destined for the tablets they have now been assigned. + +TabletServer failures are noted on the Master's monitor page, accessible via +`http://master-address:9995/monitor`. + + + +[monitor]: {{page.docs_baseurl}}/administration/overview#monitoring
http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/getting-started/shell.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/getting-started/shell.md b/_docs-unreleased/getting-started/shell.md new file mode 100644 index 0000000..7fa56d9 --- /dev/null +++ b/_docs-unreleased/getting-started/shell.md @@ -0,0 +1,145 @@ +--- +title: Accumulo Shell +category: getting-started +order: 3 +--- + +Accumulo provides a simple shell that can be used to examine the contents and +configuration settings of tables, insert/update/delete values, and change +configuration settings. + +The shell can be started by the following command: + + accumulo shell -u [username] + +The shell will prompt for the corresponding password to the username specified +and then display the following prompt: + + Shell - Apache Accumulo Interactive Shell + - + - version: 2.x.x + - instance name: myinstance + - instance id: 00000000-0000-0000-0000-000000000000 + - + - type 'help' for a list of available commands + - + root@myinstance> + +## Basic Administration + +The Accumulo shell can be used to create and delete tables, as well as to configure +table and instance specific options. + +``` +root@myinstance> tables +accumulo.metadata +accumulo.root + +root@myinstance> createtable mytable + +root@myinstance mytable> + +root@myinstance mytable> tables +accumulo.metadata +accumulo.root +mytable + +root@myinstance mytable> createtable testtable + +root@myinstance testtable> + +root@myinstance testtable> deletetable testtable +deletetable { testtable } (yes|no)? yes +Table: [testtable] has been deleted. + +root@myinstance> +``` + +The Shell can also be used to insert updates and scan tables. This is useful for +inspecting tables. + + root@myinstance mytable> scan + + root@myinstance mytable> insert row1 colf colq value1 + insert successful + + root@myinstance mytable> scan + row1 colf:colq [] value1 + +The value in brackets `[]` would be the visibility labels. Since none were used, this is empty for this row. +You can use the `-st` option to scan to see the timestamp for the cell, too. + +## Table Maintenance + +The *compact* command instructs Accumulo to schedule a compaction of the table during which +files are consolidated and deleted entries are removed. + + root@myinstance mytable> compact -t mytable + 07 16:13:53,201 [shell.Shell] INFO : Compaction of table mytable started for given range + +The *flush* command instructs Accumulo to write all entries currently in memory for a given table +to disk. + + root@myinstance mytable> flush -t mytable + 07 16:14:19,351 [shell.Shell] INFO : Flush of table mytable + initiated... + +## User Administration + +The Shell can be used to add, remove, and grant privileges to users. + +``` +root@myinstance mytable> createuser bob +Enter new password for 'bob': ********* +Please confirm new password for 'bob': ********* + +root@myinstance mytable> authenticate bob +Enter current password for 'bob': ********* +Valid + +root@myinstance mytable> grant System.CREATE_TABLE -s -u bob + +root@myinstance mytable> user bob +Enter current password for 'bob': ********* + +bob@myinstance mytable> userpermissions +System permissions: System.CREATE_TABLE +Table permissions (accumulo.metadata): Table.READ +Table permissions (mytable): NONE + +bob@myinstance mytable> createtable bobstable + +bob@myinstance bobstable> + +bob@myinstance bobstable> user root +Enter current password for 'root': ********* + +root@myinstance bobstable> revoke System.CREATE_TABLE -s -u bob +``` + +## JSR-223 Support in the Shell + +The script command can be used to invoke programs written in languages supported by installed JSR-223 +engines. You can get a list of installed engines with the -l argument. Below is an example of the output +of the command when running the Shell with Java 7. + + root@fake> script -l + Engine Alias: ECMAScript + Engine Alias: JavaScript + Engine Alias: ecmascript + Engine Alias: javascript + Engine Alias: js + Engine Alias: rhino + Language: ECMAScript (1.8) + Script Engine: Mozilla Rhino (1.7 release 3 PRERELEASE) + ScriptEngineFactory Info + +A list of compatible languages can be found on [this page](https://en.wikipedia.org/wiki/List_of_JVM_languages). The +rhino javascript engine is provided with the JVM. Typically putting a jar on the classpath is all that is +needed to install a new engine. + + When writing scripts to run in the shell, you will have a variable called connection already available +to you. This variable is a reference to an Accumulo Connector object, the same connection that the Shell +is using to communicate with the Accumulo servers. At this point you can use any of the public API methods +within your script. Reference the script command help to see all of the execution options. Script and script +invocation examples can be found in ACCUMULO-1399. http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/getting-started/table_configuration.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/getting-started/table_configuration.md b/_docs-unreleased/getting-started/table_configuration.md new file mode 100644 index 0000000..e86a1fb --- /dev/null +++ b/_docs-unreleased/getting-started/table_configuration.md @@ -0,0 +1,664 @@ +--- +title: Table Configuration +category: getting-started +order: 5 +--- + +Accumulo tables have a few options that can be configured to alter the default +behavior of Accumulo as well as improve performance based on the data stored. +These include locality groups, constraints, bloom filters, iterators, and block +cache. See the [configuration documentation][config] for a complete list of +available configuration options. + +## Locality Groups + +Accumulo supports storing sets of column families separately on disk to allow +clients to efficiently scan over columns that are frequently used together and to avoid +scanning over column families that are not requested. After a locality group is set, +Scanner and BatchScanner operations will automatically take advantage of them +whenever the fetchColumnFamilies() method is used. + +By default, tables place all column families into the same ``default'' locality group. +Additional locality groups can be configured at any time via the shell or +programmatically as follows: + +### Managing Locality Groups via the Shell + + usage: setgroups <group>=<col fam>{,<col fam>}{ <group>=<col fam>{,<col fam>}} + [-?] -t <table> + + user@myinstance mytable> setgroups group_one=colf1,colf2 -t mytable + + user@myinstance mytable> getgroups -t mytable + +### Managing Locality Groups via the Client API + +```java +Connector conn; + +HashMap<String,Set<Text>> localityGroups = new HashMap<String, Set<Text>>(); + +HashSet<Text> metadataColumns = new HashSet<Text>(); +metadataColumns.add(new Text("domain")); +metadataColumns.add(new Text("link")); + +HashSet<Text> contentColumns = new HashSet<Text>(); +contentColumns.add(new Text("body")); +contentColumns.add(new Text("images")); + +localityGroups.put("metadata", metadataColumns); +localityGroups.put("content", contentColumns); + +conn.tableOperations().setLocalityGroups("mytable", localityGroups); + +// existing locality groups can be obtained as follows +Map<String, Set<Text>> groups = + conn.tableOperations().getLocalityGroups("mytable"); +``` + +The assignment of Column Families to Locality Groups can be changed at any time. The +physical movement of column families into their new locality groups takes place via +the periodic Major Compaction process that takes place continuously in the +background. Major Compaction can also be scheduled to take place immediately +through the shell: + + user@myinstance mytable> compact -t mytable + +## Constraints + +Accumulo supports constraints applied on mutations at insert time. This can be +used to disallow certain inserts according to a user defined policy. Any mutation +that fails to meet the requirements of the constraint is rejected and sent back to the +client. + +Constraints can be enabled by setting a table property as follows: + +``` +user@myinstance mytable> constraint -t mytable -a com.test.ExampleConstraint com.test.AnotherConstraint + +user@myinstance mytable> constraint -l +com.test.ExampleConstraint=1 +com.test.AnotherConstraint=2 +``` + +Currently there are no general-purpose constraints provided with the Accumulo +distribution. New constraints can be created by writing a Java class that implements +the following interface: + + org.apache.accumulo.core.constraints.Constraint + +To deploy a new constraint, create a jar file containing the class implementing the +new constraint and place it in the lib directory of the Accumulo installation. New +constraint jars can be added to Accumulo and enabled without restarting but any +change to an existing constraint class requires Accumulo to be restarted. + +See the [contraints examples](https://github.com/apache/accumulo-examples/blob/master/docs/contraints.md) +for example code. + +## Bloom Filters + +As mutations are applied to an Accumulo table, several files are created per tablet. If +bloom filters are enabled, Accumulo will create and load a small data structure into +memory to determine whether a file contains a given key before opening the file. +This can speed up lookups considerably. + +To enable bloom filters, enter the following command in the Shell: + + user@myinstance> config -t mytable -s table.bloom.enabled=true + +The [bloom filter examples](https://github.com/apache/accumulo-examples/blob/master/docs/bloom.md) +contains an extensive example of using Bloom Filters. + +## Iterators + +Iterators provide a modular mechanism for adding functionality to be executed by +TabletServers when scanning or compacting data. This allows users to efficiently +summarize, filter, and aggregate data. In fact, the built-in features of cell-level +security and column fetching are implemented using Iterators. +Some useful Iterators are provided with Accumulo and can be found in the +*`org.apache.accumulo.core.iterators.user`* package. +In each case, any custom Iterators must be included in Accumulo's classpath, +typically by including a jar in `lib/` or `lib/ext/`, although the VFS classloader +allows for classpath manipulation using a variety of schemes including URLs and HDFS URIs. + +### Setting Iterators via the Shell + +Iterators can be configured on a table at scan, minor compaction and/or major +compaction scopes. If the Iterator implements the OptionDescriber interface, the +setiter command can be used which will interactively prompt the user to provide +values for the given necessary options. + + usage: setiter [-?] -ageoff | -agg | -class <name> | -regex | + -reqvis | -vers [-majc] [-minc] [-n <itername>] -p <pri> + [-scan] [-t <table>] + + user@myinstance mytable> setiter -t mytable -scan -p 15 -n myiter -class com.company.MyIterator + +The config command can always be used to manually configure iterators which is useful +in cases where the Iterator does not implement the OptionDescriber interface. + + config -t mytable -s table.iterator.scan.myiter=15,com.company.MyIterator + config -t mytable -s table.iterator.minc.myiter=15,com.company.MyIterator + config -t mytable -s table.iterator.majc.myiter=15,com.company.MyIterator + config -t mytable -s table.iterator.scan.myiter.opt.myoptionname=myoptionvalue + config -t mytable -s table.iterator.minc.myiter.opt.myoptionname=myoptionvalue + config -t mytable -s table.iterator.majc.myiter.opt.myoptionname=myoptionvalue + +Typically, a table will have multiple iterators. Accumulo configures a set of +system level iterators for each table. These iterators provide core +functionality like visibility label filtering and may not be removed by +users. User level iterators are applied in the order of their priority. +Priority is a user configured integer; iterators with lower numbers go first, +passing the results of their iteration on to the other iterators up the +stack. + +### Setting Iterators Programmatically + +```java +scanner.addIterator(new IteratorSetting( + 15, // priority + "myiter", // name this iterator + "com.company.MyIterator" // class name +)); +``` + +Some iterators take additional parameters from client code, as in the following +example: + +```java +IteratorSetting iter = new IteratorSetting(...); +iter.addOption("myoptionname", "myoptionvalue"); +scanner.addIterator(iter) +``` + +Tables support separate Iterator settings to be applied at scan time, upon minor +compaction and upon major compaction. For most uses, tables will have identical +iterator settings for all three to avoid inconsistent results. + +### Versioning Iterators and Timestamps + +Accumulo provides the capability to manage versioned data through the use of +timestamps within the Key. If a timestamp is not specified in the key created by the +client then the system will set the timestamp to the current time. Two keys with +identical rowIDs and columns but different timestamps are considered two versions +of the same key. If two inserts are made into Accumulo with the same rowID, +column, and timestamp, then the behavior is non-deterministic. + +Timestamps are sorted in descending order, so the most recent data comes first. +Accumulo can be configured to return the top k versions, or versions later than a +given date. The default is to return the one most recent version. + +The version policy can be changed by changing the VersioningIterator options for a +table as follows: + +``` +user@myinstance mytable> config -t mytable -s table.iterator.scan.vers.opt.maxVersions=3 + +user@myinstance mytable> config -t mytable -s table.iterator.minc.vers.opt.maxVersions=3 + +user@myinstance mytable> config -t mytable -s table.iterator.majc.vers.opt.maxVersions=3 +``` + +When a table is created, by default its configured to use the +VersioningIterator and keep one version. A table can be created without the +VersioningIterator with the -ndi option in the shell. Also the Java API +has the following method + +```java +connector.tableOperations.create(String tableName, boolean limitVersion); +``` + +#### Logical Time + +Accumulo 1.2 introduces the concept of logical time. This ensures that timestamps +set by Accumulo always move forward. This helps avoid problems caused by +TabletServers that have different time settings. The per tablet counter gives unique +one up time stamps on a per mutation basis. When using time in milliseconds, if +two things arrive within the same millisecond then both receive the same +timestamp. When using time in milliseconds, Accumulo set times will still +always move forward and never backwards. + +A table can be configured to use logical timestamps at creation time as follows: + + user@myinstance> createtable -tl logical + +#### Deletes + +Deletes are special keys in Accumulo that get sorted along will all the other data. +When a delete key is inserted, Accumulo will not show anything that has a +timestamp less than or equal to the delete key. During major compaction, any keys +older than a delete key are omitted from the new file created, and the omitted keys +are removed from disk as part of the regular garbage collection process. + +### Filters + +When scanning over a set of key-value pairs it is possible to apply an arbitrary +filtering policy through the use of a Filter. Filters are types of iterators that return +only key-value pairs that satisfy the filter logic. Accumulo has a few built-in filters +that can be configured on any table: AgeOff, ColumnAgeOff, Timestamp, NoVis, and RegEx. More can be added +by writing a Java class that extends the +`org.apache.accumulo.core.iterators.Filter` class. + +The AgeOff filter can be configured to remove data older than a certain date or a fixed +amount of time from the present. The following example sets a table to delete +everything inserted over 30 seconds ago: + +``` +user@myinstance> createtable filtertest + +user@myinstance filtertest> setiter -t filtertest -scan -minc -majc -p 10 -n myfilter -ageoff +AgeOffFilter removes entries with timestamps more than <ttl> milliseconds old +----------> set org.apache.accumulo.core.iterators.user.AgeOffFilter parameter negate, default false + keeps k/v that pass accept method, true rejects k/v that pass accept method: +----------> set org.apache.accumulo.core.iterators.user.AgeOffFilter parameter ttl, time to + live (milliseconds): 30000 +----------> set org.apache.accumulo.core.iterators.user.AgeOffFilter parameter currentTime, if set, + use the given value as the absolute time in milliseconds as the current time of day: + +user@myinstance filtertest> + +user@myinstance filtertest> scan + +user@myinstance filtertest> insert foo a b c + +user@myinstance filtertest> scan +foo a:b [] c + +user@myinstance filtertest> sleep 4 + +user@myinstance filtertest> scan + +user@myinstance filtertest> +``` + +To see the iterator settings for a table, use: + + user@example filtertest> config -t filtertest -f iterator + ---------+---------------------------------------------+------------------ + SCOPE | NAME | VALUE + ---------+---------------------------------------------+------------------ + table | table.iterator.majc.myfilter .............. | 10,org.apache.accumulo.core.iterators.user.AgeOffFilter + table | table.iterator.majc.myfilter.opt.ttl ...... | 30000 + table | table.iterator.majc.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator + table | table.iterator.majc.vers.opt.maxVersions .. | 1 + table | table.iterator.minc.myfilter .............. | 10,org.apache.accumulo.core.iterators.user.AgeOffFilter + table | table.iterator.minc.myfilter.opt.ttl ...... | 30000 + table | table.iterator.minc.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator + table | table.iterator.minc.vers.opt.maxVersions .. | 1 + table | table.iterator.scan.myfilter .............. | 10,org.apache.accumulo.core.iterators.user.AgeOffFilter + table | table.iterator.scan.myfilter.opt.ttl ...... | 30000 + table | table.iterator.scan.vers .................. | 20,org.apache.accumulo.core.iterators.VersioningIterator + table | table.iterator.scan.vers.opt.maxVersions .. | 1 + ---------+---------------------------------------------+------------------ + +### Combiners + +Accumulo supports on the fly lazy aggregation of data using Combiners. Aggregation is +done at compaction and scan time. No lookup is done at insert time, which` greatly +speeds up ingest. + +Accumulo allows Combiners to be configured on tables and column +families. When a Combiner is set it is applied across the values +associated with any keys that share rowID, column family, and column qualifier. +This is similar to the reduce step in MapReduce, which applied some function to all +the values associated with a particular key. + +For example, if a summing combiner were configured on a table and the following +mutations were inserted: + + Row Family Qualifier Timestamp Value + rowID1 colfA colqA 20100101 1 + rowID1 colfA colqA 20100102 1 + +The table would reflect only one aggregate value: + + rowID1 colfA colqA - 2 + +Combiners can be enabled for a table using the setiter command in the shell. Below is an example. + +``` +root@a14 perDayCounts> setiter -t perDayCounts -p 10 -scan -minc -majc -n daycount + -class org.apache.accumulo.core.iterators.user.SummingCombiner +TypedValueCombiner can interpret Values as a variety of number encodings + (VLong, Long, or String) before combining +----------> set SummingCombiner parameter columns, + <col fam>[:<col qual>]{,<col fam>[:<col qual>]} : day +----------> set SummingCombiner parameter type, <VARNUM|LONG|STRING>: STRING + +root@a14 perDayCounts> insert foo day 20080101 1 +root@a14 perDayCounts> insert foo day 20080101 1 +root@a14 perDayCounts> insert foo day 20080103 1 +root@a14 perDayCounts> insert bar day 20080101 1 +root@a14 perDayCounts> insert bar day 20080101 1 + +root@a14 perDayCounts> scan +bar day:20080101 [] 2 +foo day:20080101 [] 2 +foo day:20080103 [] 1 +``` + +Accumulo includes some useful Combiners out of the box. To find these look in +the *`org.apache.accumulo.core.iterators.user`* package. + +Additional Combiners can be added by creating a Java class that extends +`org.apache.accumulo.core.iterators.Combiner` and adding a jar containing that +class to Accumulo's lib/ext directory. + +See the [combiner example](https://github.com/apache/accumulo-examples/blob/master/docs/combiner.md) +for example code. + +## Block Cache + +In order to increase throughput of commonly accessed entries, Accumulo employs a block cache. +This block cache buffers data in memory so that it doesn't have to be read off of disk. +The RFile format that Accumulo prefers is a mix of index blocks and data blocks, where the index blocks are used to find the appropriate data blocks. +Typical queries to Accumulo result in a binary search over several index blocks followed by a linear scan of one or more data blocks. + +The block cache can be configured on a per-table basis, and all tablets hosted on a tablet server share a single resource pool. +To configure the size of the tablet server's block cache, set the following properties: + + tserver.cache.data.size: Specifies the size of the cache for file data blocks. + tserver.cache.index.size: Specifies the size of the cache for file indices. + +To enable the block cache for your table, set the following properties: + + table.cache.block.enable: Determines whether file (data) block cache is enabled. + table.cache.index.enable: Determines whether index cache is enabled. + +The block cache can have a significant effect on alleviating hot spots, as well as reducing query latency. +It is enabled by default for the metadata tables. + +## Compaction + +As data is written to Accumulo it is buffered in memory. The data buffered in +memory is eventually written to HDFS on a per tablet basis. Files can also be +added to tablets directly by bulk import. In the background tablet servers run +major compactions to merge multiple files into one. The tablet server has to +decide which tablets to compact and which files within a tablet to compact. +This decision is made using the compaction ratio, which is configurable on a +per table basis. To configure this ratio modify the following property: + + table.compaction.major.ratio + +Increasing this ratio will result in more files per tablet and less compaction +work. More files per tablet means more higher query latency. So adjusting +this ratio is a trade off between ingest and query performance. The ratio +defaults to 3. + +The way the ratio works is that a set of files is compacted into one file if the +sum of the sizes of the files in the set is larger than the ratio multiplied by +the size of the largest file in the set. If this is not true for the set of all +files in a tablet, the largest file is removed from consideration, and the +remaining files are considered for compaction. This is repeated until a +compaction is triggered or there are no files left to consider. + +The number of background threads tablet servers use to run major compactions is +configurable. To configure this modify the following property: + + tserver.compaction.major.concurrent.max + +Also, the number of threads tablet servers use for minor compactions is +configurable. To configure this modify the following property: + + tserver.compaction.minor.concurrent.max + +The numbers of minor and major compactions running and queued is visible on the +Accumulo monitor page. This allows you to see if compactions are backing up +and adjustments to the above settings are needed. When adjusting the number of +threads available for compactions, consider the number of cores and other tasks +running on the nodes such as maps and reduces. + +If major compactions are not keeping up, then the number of files per tablet +will grow to a point such that query performance starts to suffer. One way to +handle this situation is to increase the compaction ratio. For example, if the +compaction ratio were set to 1, then every new file added to a tablet by minor +compaction would immediately queue the tablet for major compaction. So if a +tablet has a 200M file and minor compaction writes a 1M file, then the major +compaction will attempt to merge the 200M and 1M file. If the tablet server +has lots of tablets trying to do this sort of thing, then major compactions +will back up and the number of files per tablet will start to grow, assuming +data is being continuously written. Increasing the compaction ratio will +alleviate backups by lowering the amount of major compaction work that needs to +be done. + +Another option to deal with the files per tablet growing too large is to adjust +the following property: + + table.file.max + +When a tablet reaches this number of files and needs to flush its in-memory +data to disk, it will choose to do a merging minor compaction. A merging minor +compaction will merge the tablet's smallest file with the data in memory at +minor compaction time. Therefore the number of files will not grow beyond this +limit. This will make minor compactions take longer, which will cause ingest +performance to decrease. This can cause ingest to slow down until major +compactions have enough time to catch up. When adjusting this property, also +consider adjusting the compaction ratio. Ideally, merging minor compactions +never need to occur and major compactions will keep up. It is possible to +configure the file max and compaction ratio such that only merging minor +compactions occur and major compactions never occur. This should be avoided +because doing only merging minor compactions causes O(_N_^2^) work to be done. +The amount of work done by major compactions is O(_N_*log~_R_~(_N_)) where +_R_ is the compaction ratio. + +Compactions can be initiated manually for a table. To initiate a minor +compaction, use the flush command in the shell. To initiate a major compaction, +use the compact command in the shell. The compact command will compact all +tablets in a table to one file. Even tablets with one file are compacted. This +is useful for the case where a major compaction filter is configured for a +table. In 1.4 the ability to compact a range of a table was added. To use this +feature specify start and stop rows for the compact command. This will only +compact tablets that overlap the given row range. + +### Compaction Strategies + +The default behavior of major compactions is defined in the class DefaultCompactionStrategy. +This behavior can be changed by overriding the following property with a fully qualified class name: + + table.majc.compaction.strategy + +Custom compaction strategies can have additional properties that are specified following the prefix property: + + table.majc.compaction.strategy.opts.* + +Accumulo provides a few classes that can be used as an alternative compaction strategy. These classes are located in the +org.apache.accumulo.tserver.compaction.* package. EverythingCompactionStrategy will simply compact all files. This is the +strategy used by the user "compact" command. SizeLimitCompactionStrategy compacts files no bigger than the limit set in the +property table.majc.compaction.strategy.opts.sizeLimit. + +TwoTierCompactionStrategy is a hybrid compaction strategy that supports two types of compression. If the total size of +files being compacted is larger than table.majc.compaction.strategy.opts.file.large.compress.threshold than a larger +compression type will be used. The larger compression type is specified in table.majc.compaction.strategy.opts.file.large.compress.type. +Otherwise, the configured table compression will be used. To use this strategy with minor compactions set table.file.compress.type=snappy +and set a different compress type in table.majc.compaction.strategy.opts.file.large.compress.type for larger files. + +## Pre-splitting tables + +Accumulo will balance and distribute tables across servers. Before a +table gets large, it will be maintained as a single tablet on a single +server. This limits the speed at which data can be added or queried +to the speed of a single node. To improve performance when the a table +is new, or small, you can add split points and generate new tablets. + +In the shell: + + root@myinstance> createtable newTable + root@myinstance> addsplits -t newTable g n t + +This will create a new table with 4 tablets. The table will be split +on the letters ``g'', ``n'', and ``t'' which will work nicely if the +row data start with lower-case alphabetic characters. If your row +data includes binary information or numeric information, or if the +distribution of the row information is not flat, then you would pick +different split points. Now ingest and query can proceed on 4 nodes +which can improve performance. + +## Merging tablets + +Over time, a table can get very large, so large that it has hundreds +of thousands of split points. Once there are enough tablets to spread +a table across the entire cluster, additional splits may not improve +performance, and may create unnecessary bookkeeping. The distribution +of data may change over time. For example, if row data contains date +information, and data is continually added and removed to maintain a +window of current information, tablets for older rows may be empty. + +Accumulo supports tablet merging, which can be used to reduce +the number of split points. The following command will merge all rows +from ``A'' to ``Z'' into a single tablet: + + root@myinstance> merge -t myTable -s A -e Z + +If the result of a merge produces a tablet that is larger than the +configured split size, the tablet may be split by the tablet server. +Be sure to increase your tablet size prior to any merges if the goal +is to have larger tablets: + + root@myinstance> config -t myTable -s table.split.threshold=2G + +In order to merge small tablets, you can ask Accumulo to merge +sections of a table smaller than a given size. + + root@myinstance> merge -t myTable -s 100M + +By default, small tablets will not be merged into tablets that are +already larger than the given size. This can leave isolated small +tablets. To force small tablets to be merged into larger tablets use +the `--force` option: + + root@myinstance> merge -t myTable -s 100M --force + +Merging away small tablets works on one section at a time. If your +table contains many sections of small split points, or you are +attempting to change the split size of the entire table, it will be +faster to set the split point and merge the entire table: + + root@myinstance> config -t myTable -s table.split.threshold=256M + root@myinstance> merge -t myTable + +## Delete Range + +Consider an indexing scheme that uses date information in each row. +For example ``20110823-15:20:25.013'' might be a row that specifies a +date and time. In some cases, we might like to delete rows based on +this date, say to remove all the data older than the current year. +Accumulo supports a delete range operation which efficiently +removes data between two rows. For example: + + root@myinstance> deleterange -t myTable -s 2010 -e 2011 + +This will delete all rows starting with ``2010'' and it will stop at +any row starting ``2011''. You can delete any data prior to 2011 +with: + + root@myinstance> deleterange -t myTable -e 2011 --force + +The shell will not allow you to delete an unbounded range (no start) +unless you provide the `--force` option. + +Range deletion is implemented using splits at the given start/end +positions, and will affect the number of splits in the table. + +## Cloning Tables + +A new table can be created that points to an existing table's data. This is a +very quick metadata operation, no data is actually copied. The cloned table +and the source table can change independently after the clone operation. One +use case for this feature is testing. For example to test a new filtering +iterator, clone the table, add the filter to the clone, and force a major +compaction. To perform a test on less data, clone a table and then use delete +range to efficiently remove a lot of data from the clone. Another use case is +generating a snapshot to guard against human error. To create a snapshot, +clone a table and then disable write permissions on the clone. + +The clone operation will point to the source table's files. This is why the +flush option is present and is enabled by default in the shell. If the flush +option is not enabled, then any data the source table currently has in memory +will not exist in the clone. + +A cloned table copies the configuration of the source table. However the +permissions of the source table are not copied to the clone. After a clone is +created, only the user that created the clone can read and write to it. + +In the following example we see that data inserted after the clone operation is +not visible in the clone. + +``` +root@a14> createtable people + +root@a14 people> insert 890435 name last Doe +root@a14 people> insert 890435 name first John + +root@a14 people> clonetable people test + +root@a14 people> insert 890436 name first Jane +root@a14 people> insert 890436 name last Doe + +root@a14 people> scan +890435 name:first [] John +890435 name:last [] Doe +890436 name:first [] Jane +890436 name:last [] Doe + +root@a14 people> table test + +root@a14 test> scan +890435 name:first [] John +890435 name:last [] Doe + +root@a14 test> +``` + +The du command in the shell shows how much space a table is using in HDFS. +This command can also show how much overlapping space two cloned tables have in +HDFS. In the example below du shows table ci is using 428M. Then ci is cloned +to cic and du shows that both tables share 428M. After three entries are +inserted into cic and its flushed, du shows the two tables still share 428M but +cic has 226 bytes to itself. Finally, table cic is compacted and then du shows +that each table uses 428M. + +``` +root@a14> du ci + 428,482,573 [ci] + +root@a14> clonetable ci cic + +root@a14> du ci cic + 428,482,573 [ci, cic] + +root@a14> table cic + +root@a14 cic> insert r1 cf1 cq1 v1 +root@a14 cic> insert r1 cf1 cq2 v2 +root@a14 cic> insert r1 cf1 cq3 v3 + +root@a14 cic> flush -t cic -w +27 15:00:13,908 [shell.Shell] INFO : Flush of table cic completed. + +root@a14 cic> du ci cic + 428,482,573 [ci, cic] + 226 [cic] + +root@a14 cic> compact -t cic -w +27 15:00:35,871 [shell.Shell] INFO : Compacting table ... +27 15:03:03,303 [shell.Shell] INFO : Compaction of table cic completed for given range + +root@a14 cic> du ci cic + 428,482,573 [ci] + 428,482,612 [cic] + +root@a14 cic> +``` + +## Exporting Tables + +Accumulo supports exporting tables for the purpose of copying tables to another +cluster. Exporting and importing tables preserves the tables configuration, +splits, and logical time. Tables are exported and then copied via the hadoop +distcp command. To export a table, it must be offline and stay offline while +discp runs. The reason it needs to stay offline is to prevent files from being +deleted. A table can be cloned and the clone taken offline inorder to avoid +losing access to the table. See the [export example](https://github.com/apache/accumulo-examples/blob/master/docs/export.md) +for example code. + +[config]: /docs/{{ page.version }}/config/ http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/getting-started/table_design.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/getting-started/table_design.md b/_docs-unreleased/getting-started/table_design.md new file mode 100644 index 0000000..fb547db --- /dev/null +++ b/_docs-unreleased/getting-started/table_design.md @@ -0,0 +1,303 @@ +--- +title: Table Design +category: getting-started +order: 4 +--- + +### Basic Table + +Since Accumulo tables are sorted by row ID, each table can be thought of as being +indexed by the row ID. Lookups performed by row ID can be executed quickly, by doing +a binary search, first across the tablets, and then within a tablet. Clients should +choose a row ID carefully in order to support their desired application. A simple rule +is to select a unique identifier as the row ID for each entity to be stored and assign +all the other attributes to be tracked to be columns under this row ID. For example, +if we have the following data in a comma-separated file: + + userid,age,address,account-balance + +We might choose to store this data using the userid as the rowID, the column +name in the column family, and a blank column qualifier: + +```java +Mutation m = new Mutation(userid); +final String column_qualifier = ""; +m.put("age", column_qualifier, age); +m.put("address", column_qualifier, address); +m.put("balance", column_qualifier, account_balance); + +writer.add(m); +``` + +We could then retrieve any of the columns for a specific userid by specifying the +userid as the range of a scanner and fetching specific columns: + +```java +Range r = new Range(userid, userid); // single row +Scanner s = conn.createScanner("userdata", auths); +s.setRange(r); +s.fetchColumnFamily(new Text("age")); + +for(Entry<Key,Value> entry : s) { + System.out.println(entry.getValue().toString()); +} +``` + +### RowID Design + +Often it is necessary to transform the rowID in order to have rows ordered in a way +that is optimal for anticipated access patterns. A good example of this is reversing +the order of components of internet domain names in order to group rows of the +same parent domain together: + + com.google.code + com.google.labs + com.google.mail + com.yahoo.mail + com.yahoo.research + +Some data may result in the creation of very large rows - rows with many columns. +In this case the table designer may wish to split up these rows for better load +balancing while keeping them sorted together for scanning purposes. This can be +done by appending a random substring at the end of the row: + + com.google.code_00 + com.google.code_01 + com.google.code_02 + com.google.labs_00 + com.google.mail_00 + com.google.mail_01 + +It could also be done by adding a string representation of some period of time such as date to the week +or month: + + com.google.code_201003 + com.google.code_201004 + com.google.code_201005 + com.google.labs_201003 + com.google.mail_201003 + com.google.mail_201004 + +Appending dates provides the additional capability of restricting a scan to a given +date range. + +### Lexicoders +Since Keys in Accumulo are sorted lexicographically by default, it's often useful to encode +common data types into a byte format in which their sort order corresponds to the sort order +in their native form. An example of this is encoding dates and numerical data so that they can +be better seeked or searched in ranges. + +The lexicoders are a standard and extensible way of encoding Java types. Here's an example +of a lexicoder that encodes a java Date object so that it sorts lexicographically: + +```java +// create new date lexicoder +DateLexicoder dateEncoder = new DateLexicoder(); + +// truncate time to hours +long epoch = System.currentTimeMillis(); +Date hour = new Date(epoch - (epoch % 3600000)); + +// encode the rowId so that it is sorted lexicographically +Mutation mutation = new Mutation(dateEncoder.encode(hour)); +mutation.put(new Text("colf"), new Text("colq"), new Value(new byte[]{})); +``` + +If we want to return the most recent date first, we can reverse the sort order +with the reverse lexicoder: + +```java +// create new date lexicoder and reverse lexicoder +DateLexicoder dateEncoder = new DateLexicoder(); +ReverseLexicoder reverseEncoder = new ReverseLexicoder(dateEncoder); + +// truncate date to hours +long epoch = System.currentTimeMillis(); +Date hour = new Date(epoch - (epoch % 3600000)); + +// encode the rowId so that it sorts in reverse lexicographic order +Mutation mutation = new Mutation(reverseEncoder.encode(hour)); +mutation.put(new Text("colf"), new Text("colq"), new Value(new byte[]{})); +``` + +### Indexing + +In order to support lookups via more than one attribute of an entity, additional +indexes can be built. However, because Accumulo tables can support any number of +columns without specifying them beforehand, a single additional index will often +suffice for supporting lookups of records in the main table. Here, the index has, as +the rowID, the Value or Term from the main table, the column families are the same, +and the column qualifier of the index table contains the rowID from the main table. + +| RowID | Column Family | Column Qualifier | Value | +|-------|---------------|------------------|-------| +| Term | Field Name | MainRowID | | + +Note: We store rowIDs in the column qualifier rather than the Value so that we can +have more than one rowID associated with a particular term within the index. If we +stored this in the Value we would only see one of the rows in which the value +appears since Accumulo is configured by default to return the one most recent +value associated with a key. + +Lookups can then be done by scanning the Index Table first for occurrences of the +desired values in the columns specified, which returns a list of row ID from the main +table. These can then be used to retrieve each matching record, in their entirety, or a +subset of their columns, from the Main Table. + +To support efficient lookups of multiple rowIDs from the same table, the Accumulo +client library provides a BatchScanner. Users specify a set of Ranges to the +BatchScanner, which performs the lookups in multiple threads to multiple servers +and returns an Iterator over all the rows retrieved. The rows returned are NOT in +sorted order, as is the case with the basic Scanner interface. + +```java +// first we scan the index for IDs of rows matching our query +Text term = new Text("mySearchTerm"); + +HashSet<Range> matchingRows = new HashSet<Range>(); + +Scanner indexScanner = createScanner("index", auths); +indexScanner.setRange(new Range(term, term)); + +// we retrieve the matching rowIDs and create a set of ranges +for(Entry<Key,Value> entry : indexScanner) { + matchingRows.add(new Range(entry.getKey().getColumnQualifier())); +} + +// now we pass the set of rowIDs to the batch scanner to retrieve them +BatchScanner bscan = conn.createBatchScanner("table", auths, 10); +bscan.setRanges(matchingRows); +bscan.fetchColumnFamily(new Text("attributes")); + +for(Entry<Key,Value> entry : bscan) { + System.out.println(entry.getValue()); +} +``` + +One advantage of the dynamic schema capabilities of Accumulo is that different +fields may be indexed into the same physical table. However, it may be necessary to +create different index tables if the terms must be formatted differently in order to +maintain proper sort order. For example, real numbers must be formatted +differently than their usual notation in order to be sorted correctly. In these cases, +usually one index per unique data type will suffice. + +### Entity-Attribute and Graph Tables + +Accumulo is ideal for storing entities and their attributes, especially of the +attributes are sparse. It is often useful to join several datasets together on common +entities within the same table. This can allow for the representation of graphs, +including nodes, their attributes, and connections to other nodes. + +Rather than storing individual events, Entity-Attribute or Graph tables store +aggregate information about the entities involved in the events and the +relationships between entities. This is often preferrable when single events aren't +very useful and when a continuously updated summarization is desired. + +The physical schema for an entity-attribute or graph table is as follows: + +|RowID |Column Family |Column Qualifier |Value | +|---------|---------------|-----------------|------| +|EntityID |Attribute Name |Attribute Value |Weight| +|EntityID |Edge Type |Related EntityID |Weight| + +For example, to keep track of employees, managers and products the following +entity-attribute table could be used. Note that the weights are not always necessary +and are set to 0 when not used. + +|RowID |Column Family |Column Qualifier |Value +|------|--------------|-----------------|----- +| E001 | name | bob | 0 +| E001 | department | sales | 0 +| E001 | hire_date | 20030102 | 0 +| E001 | units_sold | P001 | 780 +| E002 | name | george | 0 +| E002 | department | sales | 0 +| E002 | manager_of | E001 | 0 +| E002 | manager_of | E003 | 0 +| E003 | name | harry | 0 +| E003 | department | accounts_recv | 0 +| E003 | hire_date | 20000405 | 0 +| E003 | units_sold | P002 | 566 +| E003 | units_sold | P001 | 232 +| P001 | product_name | nike_airs | 0 +| P001 | product_type | shoe | 0 +| P001 | in_stock | germany | 900 +| P001 | in_stock | brazil | 200 +| P002 | product_name | basic_jacket | 0 +| P002 | product_type | clothing | 0 +| P002 | in_stock | usa | 3454 +| P002 | in_stock | germany | 700 + +To allow efficient updating of edge weights, an aggregating iterator can be +configured to add the value of all mutations applied with the same key. These types +of tables can easily be created from raw events by simply extracting the entities, +attributes, and relationships from individual events and inserting the keys into +Accumulo each with a count of 1. The aggregating iterator will take care of +maintaining the edge weights. + +### Document-Partitioned Indexing + +Using a simple index as described above works well when looking for records that +match one of a set of given criteria. When looking for records that match more than +one criterion simultaneously, such as when looking for documents that contain all of +the words 'the' and 'white' and 'house', there are several issues. + +First is that the set of all records matching any one of the search terms must be sent +to the client, which incurs a lot of network traffic. The second problem is that the +client is responsible for performing set intersection on the sets of records returned +to eliminate all but the records matching all search terms. The memory of the client +may easily be overwhelmed during this operation. + +For these reasons Accumulo includes support for a scheme known as sharded +indexing, in which these set operations can be performed at the TabletServers and +decisions about which records to include in the result set can be made without +incurring network traffic. + +This is accomplished via partitioning records into bins that each reside on at most +one TabletServer, and then creating an index of terms per record within each bin as +follows: + +|RowID |Column Family |Column Qualifier |Value +|------|--------------|-----------------|------ +|BinID |Term |DocID |Weight + +Documents or records are mapped into bins by a user-defined ingest application. By +storing the BinID as the RowID we ensure that all the information for a particular +bin is contained in a single tablet and hosted on a single TabletServer since +Accumulo never splits rows across tablets. Storing the Terms as column families +serves to enable fast lookups of all the documents within this bin that contain the +given term. + +Finally, we perform set intersection operations on the TabletServer via a special +iterator called the Intersecting Iterator. Since documents are partitioned into many +bins, a search of all documents must search every bin. We can use the BatchScanner +to scan all bins in parallel. The Intersecting Iterator should be enabled on a +BatchScanner within user query code as follows: + +```java +Text[] terms = {new Text("the"), new Text("white"), new Text("house")}; + +BatchScanner bscan = conn.createBatchScanner(table, auths, 20); + +IteratorSetting iter = new IteratorSetting(20, "ii", IntersectingIterator.class); +IntersectingIterator.setColumnFamilies(iter, terms); + +bscan.addScanIterator(iter); +bscan.setRanges(Collections.singleton(new Range())); + +for(Entry<Key,Value> entry : bscan) { + System.out.println(" " + entry.getKey().getColumnQualifier()); +} +``` + +This code effectively has the BatchScanner scan all tablets of a table, looking for +documents that match all the given terms. Because all tablets are being scanned for +every query, each query is more expensive than other Accumulo scans, which +typically involve a small number of TabletServers. This reduces the number of +concurrent queries supported and is subject to what is known as the 'straggler' +problem in which every query runs as slow as the slowest server participating. + +Of course, fast servers will return their results to the client which can display them +to the user immediately while they wait for the rest of the results to arrive. If the +results are unordered this is quite effective as the first results to arrive are as good +as any others to the user. http://git-wip-us.apache.org/repos/asf/accumulo-website/blob/7cc70b2e/_docs-unreleased/index.md ---------------------------------------------------------------------- diff --git a/_docs-unreleased/index.md b/_docs-unreleased/index.md new file mode 100644 index 0000000..02e840c --- /dev/null +++ b/_docs-unreleased/index.md @@ -0,0 +1,16 @@ +--- +title: Apache Accumulo documentation +nodoctitle: true +--- + +# Accumulo {{ page.version }} documentation + +Welcome to the documentation for the {{ page.version }} release of Apache Accumulo. + +Use the sidebar to left the navigate the documentation. + +The links below contain additional documentation for this release: + +* [Release notes](/release/accumulo-1.8.1) +* [Javadocs](/1.8/apidocs) +* [Examples](/1.8/examples)
