Author: bhavanki
Date: Thu May 1 21:02:38 2014
New Revision: 1591779
URL: http://svn.apache.org/r1591779
Log:
Merged trunk into redesign14 branch - bylaw update, glossary update, relnotes
1.6.0 update.
Modified:
accumulo/site/branches/redesign14/ (props changed)
accumulo/site/branches/redesign14/content/bylaws.mdtext
accumulo/site/branches/redesign14/content/glossary.mdtext
accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext
Propchange: accumulo/site/branches/redesign14/
------------------------------------------------------------------------------
Merged /accumulo/site/trunk:r1587699-1591778
Modified: accumulo/site/branches/redesign14/content/bylaws.mdtext
URL:
http://svn.apache.org/viewvc/accumulo/site/branches/redesign14/content/bylaws.mdtext?rev=1591779&r1=1591778&r2=1591779&view=diff
==============================================================================
--- accumulo/site/branches/redesign14/content/bylaws.mdtext (original)
+++ accumulo/site/branches/redesign14/content/bylaws.mdtext Thu May 1 21:02:38
2014
@@ -17,7 +17,7 @@ Notice: Licensed to the Apache Softwa
specific language governing permissions and limitations
under the License.
-This is version 2 of the bylaws. Community work actively continues on the
bylaws, and so key segments of them are subject to change.
+This is version 3 of the bylaws. Community work actively continues on the
bylaws, and so key segments of them are subject to change.
# Introduction
@@ -210,7 +210,7 @@ For Code Change actions, a committer may
<td>3</td>
<tr><td>Modifying Bylaws</td>
<td>Modifying this document.</td>
- <td>Majority approval</td>
+ <td>Consensus approval</td>
<td>Active PMC members</td>
<td>7</td>
</table>
Modified: accumulo/site/branches/redesign14/content/glossary.mdtext
URL:
http://svn.apache.org/viewvc/accumulo/site/branches/redesign14/content/glossary.mdtext?rev=1591779&r1=1591778&r2=1591779&view=diff
==============================================================================
--- accumulo/site/branches/redesign14/content/glossary.mdtext (original)
+++ accumulo/site/branches/redesign14/content/glossary.mdtext Thu May 1
21:02:38 2014
@@ -37,7 +37,7 @@ Notice: Licensed to the Apache Softwa
<dt>iterator scopes</dt>
<dd>the possible scopes for iterators are where the tablet server is already
reading and/or writing data: minor compaction / flush time (<em>minc</em>
scope), major compaction / file merging time (<em>majc</em> scope), and query
time (<em>scan</em> scope).</dd>
<dt>gc</dt>
-<dd> </dd>
+<dd>process that identifies temporary files in HDFS that are no longer needed
by any process, and deletes them.</dd>
<dt>key</dt>
<dd>the key into the distributed sorted map which is accumulo. The key is
subdivided into row, column, and timestamp. The column is further divided into
family, qualifier, and visibility.</dd>
<dt>locality group</dt>
@@ -45,19 +45,19 @@ Notice: Licensed to the Apache Softwa
<dt>log-structured merge-tree</dt>
<dd>the sorting / flushing / merging scheme on which BigTable's design is
based.</dd>
<dt>logger</dt>
-<dd> </dd>
+<dd>in 1.4 and older, process that accepts updates to tablet servers and
writes them to local on-disk storage for redundancy. in 1.5 the functionality
was subsumed by the tablet server and datanode with HDFS writes.</dd>
<dt>major compaction</dt>
<dd>merging multiple files into a single file. If all of a tablet's files are
merged into a single file, it is called a <em>full major compaction</em>.</dd>
<dt>master</dt>
-<dd> </dd>
+<dd>process that detects and responds to tablet failures, balances load across
tablet servers by assigning and migrating tablets when required, coordinates
table operations, and handles tablet server logistics (startup, shutdown,
recovery).</dd>
<dt>minor compaction</dt>
<dd>flushing data from memory to disk. Usually this creates a new file for a
tablet, but if the memory flushed is merge-sorted in with data from an existing
file (replacing that file), it is called a <em>merging minor
compaction</em>.</dd>
<dt>monitor</dt>
-<dd> </dd>
+<dd>process that displays status and usage information for all Accumulo
components.</dd>
<dt>permissions</dt>
<dd>administrative abilities that must be given to a user such as creating
tables or users and changing permissions or configuration parameters.</dd>
<dt>row</dt>
-<dd>the portion of the key that is controls atomicity. Keys with the same row
are guaranteed to remain on a single tablet hosted by a single tablet server,
therefore multiple key/value pairs can be added to or removed from a row at the
same time. The row is used for the primary sorting of the key.</dd>
+<dd>the portion of the key that controls atomicity. Keys with the same row
are guaranteed to remain on a single tablet hosted by a single tablet server,
therefore multiple key/value pairs can be added to or removed from a row at the
same time. The row is used for the primary sorting of the key.</dd>
<dt>scan</dt>
<dd>reading a range of key/value pairs.</dd>
<dt>tablet</dt>
@@ -68,4 +68,5 @@ Notice: Licensed to the Apache Softwa
<dd>the portion of the key that controls versioning. Otherwise identical keys
with differing timestamps are considered to be versions of a single
<em>cell</em>. Accumulo can be configured to keep the <em>N</em> newest
versions of each <em>cell</em>. When a deletion entry is inserted, it deletes
all earlier versions for its cell.</dd>
<dt>value</dt>
<dd>immutable bytes associated with a particular key.</dd>
-</dl>
\ No newline at end of file
+</dl>
+
Modified: accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext
URL:
http://svn.apache.org/viewvc/accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext?rev=1591779&r1=1591778&r2=1591779&view=diff
==============================================================================
--- accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext
(original)
+++ accumulo/site/branches/redesign14/content/release_notes/1.6.0.mdtext Thu
May 1 21:02:38 2014
@@ -1,4 +1,4 @@
-Title:
+Title: Release Notes for Apache Accumulo 1.6.0
Notice: Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -24,7 +24,7 @@ Accumulo 1.6.0 runs on Hadoop 1, however
## Notable Improvements
-### Multiple namenode support
+### Multiple volume support
[BigTable's][1] design allows for its internal metadata to automatically
spread across multiple nodes. Accumulo has followed this design and scales
very well as a result. There is one impediment to scaling though, and this is
the HDFS namenode. There are two problems with the namenode when it comes to
scaling. First, the namenode stores all of its filesystem metadata in memory
on a single machine. This introduces an upper bound on the number of files
Accumulo can have. Second, there is an upper bound on the number of file
operations per second that a single namenode can support. For example, a
namenode can only support a few thousand delete or create file request per
second.
@@ -46,7 +46,9 @@ The only built in test that conditional
### Encryption
-Support for encrypting Accumulo's persistent and over the wire data was added.
[ACCUMULO-998][ACCUMULO-998], [ACCUMULO-958][ACCUMULO-958], and
[ACCUMULO-980][ACCUMULO-980] cover encrypting data at rest in write ahead logs
and rfiles. [ACCUMULO-1009][ACCUMULO-1009] covers encrypting data over the
wire using SSL.
+Encryption is still an experimental feature, but much progress has been made
since 1.5.0. Support for encrypting rfiles and write ahead logs were added in
[ACCUMULO-958][ACCUMULO-958] and [ACCUMULO-980][ACCUMULO-980]. Support for
encrypting data over the wire using SSL was added in
[ACCUMULO-1009][ACCUMULO-1009].
+
+When a tablet server fails, its write ahead logs are sorted and stored in
HDFS. In 1.6.0, encrypting these sorted write ahead logs is not supported.
[ACCUMULO-981][ACCUMULO-981] addresses this issue.
### Pluggable compaction strategies
@@ -82,6 +84,42 @@ While Hadoop [does not support IPv6 netw
Multiple bug-fixes were made to support running Accumulo over multiple HDFS
instances using ViewFS. [ACCUMULO-2047][ACCUMULO-2047] is the parent
ticket that contains numerous fixes to enable this support.
+### Maven Plugin
+
+This version of Accumulo is accompanied by a new maven plugin for testing
client apps ([ACCUMULO-1030][ACCUMULO-1030]). You can execute the
accumulo-maven-plugin inside your project by adding the following to your
pom.xml's build plugins section:
+
+ <plugin>
+ <groupId>org.apache.accumulo</groupId>
+ <artifactId>accumulo-maven-plugin</artifactId>
+ <version>1.6.0</version>
+ <configuration>
+ <instanceName>plugin-it-instance</instanceName>
+ <rootPassword>ITSecret</rootPassword>
+ </configuration>
+ <executions>
+ <execution>
+ <id>run-plugin</id>
+ <goals>
+ <goal>start</goal>
+ <goal>stop</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+
+This plugin is designed to work in conjunction with the maven-failsafe-plugin.
A small test instance of Accumulo will run during the pre-integration-test
phase of the Maven build lifecycle, and will be stopped in the
post-integration-test phase. Your integration tests, executed by
maven-failsafe-plugin can access this instance with a MiniAccumuloInstance
connector (the plugin uses MiniAccumuloInstance, internally), as in the
following example:
+
+ private static Connector conn;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ String instanceName = "plugin-it-instance";
+ Instance instance = new MiniAccumuloInstance(instanceName, new
File("target/accumulo-maven-plugin/" + instanceName));
+ conn = instance.getConnector("root", new PasswordToken("ITSecret"));
+ }
+
+This plugin is quite limited, currently only supporting an instance name and a
root user password as configuration parameters. Improvements are expected in
future releases, so feedback is welcome and appreciated (file bugs/requests
under the "maven-plugin" component in the Accumulo JIRA).
+
### Packaging
One notable change that was made to the binary tarball is the purposeful
omission of a pre-built copy of the Accumulo "native map" library.
@@ -94,6 +132,12 @@ It is recommended that users invoke the
Be aware that you will need a C++ compiler/toolchain installed to build this
library. Check your GNU/Linux distribution documentation for the package
manager command.
+#### Size-Based Constraint on New Tables
+
+A Constraint is an interface that can determine if a Mutation should be
applied or rejected server-side. New tables that are created in 1.6.0 will
automatically have the `DefaultKeySizeConstraint` set.
+As performance can suffer when large Keys are inserted into a table, this
Constraint will reject any Key that is larger than 1MB. If this constraint is
undesired, it can be removed using the `constraint` shell
+command. See the help message on the command for more information.
+
### Other notable changes
* [ACCUMULO-842][ACCUMULO-842] Added FATE administration to shell
@@ -132,6 +176,8 @@ Be aware that you will need a C++ compil
## Known Issues
+### Slower writes than previous Accumulo versions
+
When using Accumulo 1.6 and Hadoop 2, Accumulo will call hsync() on HDFS.
Calling hsync improves durability by ensuring data is on disk (where other
older
Hadoop versions might lose data in the face of power failure); however, calling
@@ -145,8 +191,20 @@ mutation queues.
For more information, see [ACCUMULO-1950][ACCUMULO-1950] and [this
comment][ACCUMULO-1905-comment].
+Another possible cause of slower writes is the change in write ahead log
replication
+between 1.4 and 1.5. Accumulo 1.4. defaulted to two loggers servers.
Accumulo 1.5 and 1.6 store
+write ahead logs in HDFS and default to using three datanodes.
+
+### BatchWriter hold time error
+
+If a `BatchWriter` fails with `MutationsRejectedException` and the message
contains
+`"# server errors 1"` then it may be [ACCUMULO-2388][ACCUMULO-2388]. To
confirm this look in the tablet server logs
+for `org.apache.accumulo.tserver.HoldTimeoutException` around the time the
`BatchWriter` failed.
+If this is happening often a possible work around is to set
`general.rpc.timeout` to `240s`.
+
### Other known issues
+ * [ACCUMULO-981][ACCUMULO-981] Sorted write ahead logs are not encrypted.
* [ACCUMULO-1507][ACCUMULO-1507] Dynamic Classloader still can't keep proper
track of jars
* [ACCUMULO-1588][ACCUMULO-1588] Monitor XML and JSON differ
* [ACCUMULO-1628][ACCUMULO-1628] NPE on deep copied dumped memory iterator
@@ -214,13 +272,14 @@ and, in HDFS High-Availability instances
[ACCUMULO-802]: https://issues.apache.org/jira/browse/ACCUMULO-802 "Table
namespaces"
[ACCUMULO-842]: https://issues.apache.org/jira/browse/ACCUMULO-842 "Add FATE
administration to shell"
[ACCUMULO-958]: https://issues.apache.org/jira/browse/ACCUMULO-958 "Support
pluggable encryption in walogs"
-[ACCUMULO-998]: https://issues.apache.org/jira/browse/ACCUMULO-998 "Support
encryption at rest"
[ACCUMULO-970]: https://issues.apache.org/jira/browse/ACCUMULO-970
[ACCUMULO-980]: https://issues.apache.org/jira/browse/ACCUMULO-980 "Support
pluggable codecs for RFile"
+[ACCUMULO-981]: https://issues.apache.org/jira/browse/ACCUMULO-981 "support
pluggable encryption when recovering write-ahead logs"
[ACCUMULO-1000]: https://issues.apache.org/jira/browse/ACCUMULO-1000
"Conditional Mutations"
[ACCUMULO-1009]: https://issues.apache.org/jira/browse/ACCUMULO-1009 "Support
encryption over the wire"
[ACCUMULO-1018]: https://issues.apache.org/jira/browse/ACCUMULO-1018 "Client
does not give informative message when user can not read table"
[ACCUMULO-1024]: https://issues.apache.org/jira/browse/ACCUMULO-1024
+[ACCUMULO-1030]: https://issues.apache.org/jira/browse/ACCUMULO-1030 "Create a
Maven plugin to run MiniAccumuloCluster for integration testing"
[ACCUMULO-1218]: https://issues.apache.org/jira/browse/ACCUMULO-1218 "document
the recovery from a failed zookeeper"
[ACCUMULO-1336]: https://issues.apache.org/jira/browse/ACCUMULO-1336 "Add
lexicoders from Typo to Accumulo"
[ACCUMULO-1375]: https://issues.apache.org/jira/browse/ACCUMULO-1375 "Update
README files in proxy module."
@@ -270,6 +329,7 @@ and, in HDFS High-Availability instances
[ACCUMULO-2262]: https://issues.apache.org/jira/browse/ACCUMULO-2262 "Include
java.net.preferIPv4Stack=true in process startup"
[ACCUMULO-2334]: https://issues.apache.org/jira/browse/ACCUMULO-2334 "Lacking
fallback when ACCUMULO_LOG_HOST isn't set"
[ACCUMULO-2360]: https://issues.apache.org/jira/browse/ACCUMULO-2360 "Need a
way to configure TNonblockingServer.maxReadBufferBytes to prevent OOMs"
+[ACCUMULO-2388]: https://issues.apache.org/jira/browse/ACCUMULO-2388
[ACCUMULO-2408]: https://issues.apache.org/jira/browse/ACCUMULO-2408 "metadata
table not assigned after root table is loaded"
[ACCUMULO-2441]: https://issues.apache.org/jira/browse/ACCUMULO-2441 "Document
internal state stored in RFile names"
[ACCUMULO-2495]: https://issues.apache.org/jira/browse/ACCUMULO-2495 "OOM
exception didn't bring down tserver"
