Author: buildbot
Date: Fri Oct 3 16:22:57 2014
New Revision: 924532
Log:
Staging update by buildbot for accumulo
Modified:
websites/staging/accumulo/trunk/content/ (props changed)
websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html
Propchange: websites/staging/accumulo/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Fri Oct 3 16:22:57 2014
@@ -1 +1 @@
-1627921
+1629249
Modified: websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html
==============================================================================
--- websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html
(original)
+++ websites/staging/accumulo/trunk/content/1.6/accumulo_user_manual.html Fri
Oct 3 16:22:57 2014
@@ -548,6 +548,8 @@ span.icon > [class^="icon-"], span.icon
<li><a href="#_cluster_specification">11.5.3. Cluster Specification</a></li>
<li><a href="#_accumulo_settings">11.5.4. Accumulo Settings</a></li>
<li><a href="#_deploy_configuration">11.5.5. Deploy Configuration</a></li>
+<li><a href="#_sensitive_configuration_values">11.5.6 Sensitive Configuration
Values</a></li>
+<li><a href="#_using_a_javakeystorecredentialprovider_for_storage">11.5.7.
Using a JavaKeyStoreCredentialProvider for storage</a></li>
</ul>
</li>
<li><a href="#_initialization">11.6. Initialization</a></li>
@@ -597,7 +599,8 @@ span.icon > [class^="icon-"], span.icon
<li><a href="#zookeeper_failure">13.8.2. ZooKeeper Failure</a></li>
</ul>
</li>
-<li><a href="#_file_naming_conventions">13.9. File Naming Conventions</a></li>
+<li><a href="#_upgrade_issues">13.9. Upgrade Issues</a></li>
+<li><a href="#_file_naming_conventions">13.10. File Naming Conventions</a></li>
</ul>
</li>
<li><a href="#configuration">14. Appendix A: Configuration Management</a></li>
@@ -1296,6 +1299,17 @@ Instance inst = new ZooKeeperInstance(in
Connector conn = inst.getConnector("user", new
PasswordToken("passwd"));</code></pre>
</div>
</div>
+<div class="paragraph">
+<p>The PasswordToken is the most common implementation of an
<code>AuthenticationToken</code>.
+This general interface allows authentication as an Accumulo user to come from
a variety of sources
+or means. The CredentialProviderToken leverages the Hadoop CredentialProviders
(new in Hadoop 2.6).</p>
+</div>
+<div class="paragraph">
+<p>For example, the CredentialProviderToken can be used in conjunction with a
Java KeyStore
+to alleviate passwords stored in cleartext. When stored in HDFS, a single
KeyStore can be
+used across an entire instance. Be aware that KeyStores stored on the local
filesystem
+must be made available to all nodes in the Accumulo cluster.</p>
+</div>
</div>
<div class="sect2">
<h3 id="_writing_data">4.3. Writing Data</h3>
@@ -1405,10 +1419,10 @@ Scanner scan =
conn.createScanner("table", auths);
scan.setRange(new Range("harry","john"));
-scan.fetchFamily("attributes");
+scan.fetchColumnFamily(new Text("attributes"));
for(Entry<Key,Value> entry : scan) {
- String row = entry.getKey().getRow();
+ Text row = entry.getKey().getRow();
Value value = entry.getValue();
}</code></pre>
</div>
@@ -1478,7 +1492,7 @@ TabletServers in parallel.</p>
BatchScanner bscan =
conn.createBatchScanner("table", auths, 10);
bscan.setRanges(ranges);
-bscan.fetchFamily("attributes");
+bscan.fetchColumnFamily(new Text("attributes"));
for(Entry<Key,Value> entry : scan) {
System.out.println(entry.getValue());
@@ -1944,6 +1958,14 @@ config -t mytable -s table.iterator.minc
config -t mytable -s
table.iterator.majc.myiter.opt.myoptionname=myoptionvalue</pre>
</div>
</div>
+<div class="paragraph">
+<p>Typically, a table will have multiple iterators. Accumulo configures a set
of
+system level iterators for each table. These iterators provide core
functionality
+like visibility label filtering and may not be removed by users. User level
iterators
+are applied in the order of their priority. Priority is a user-configured
integer;
+iterators with lower numbers go first, passing the results of their iteration
on
+to the other iterators up the stack.</p>
+</div>
</div>
<div class="sect3">
<h4 id="_setting_iterators_programmatically">6.4.2. Setting Iterators
Programmatically</h4>
@@ -3569,8 +3591,14 @@ mutation.put(colFam, colQual, colVis, ti
<div class="paragraph">
<p>Security labels consist of a set of user-defined tokens that are required
to read the
value the label is associated with. The set of tokens required can be
specified using
-syntax that supports logical AND and OR combinations of tokens, as well as
nesting
-groups of tokens together.</p>
+syntax that supports logical AND <code>&</code> and OR <code>|</code>
combinations of tokens, as well as nesting
+groups <code>()</code> of tokens together.</p>
+</div>
+<div class="paragraph">
+<p>Each term is comprised of one to many alpha-numeric characters, hyphens,
underscores
+or periods. Optionally, each term may be wrapped in quotation marks which
removes the
+restriction on valid characters. In quoted terms, quotation marks and
backslash characters
+can be used as characters in the term by escaping them with a backslash.</p>
</div>
<div class="paragraph">
<p>For example, suppose within our organization we want to label our data
values with
@@ -3926,17 +3954,20 @@ native map also avoids the performance i
in the JVM by causing it to pause much less frequently.</p>
</div>
<div class="paragraph">
-<p>32-bit and 64-bit Linux versions of the native map ship with the Accumulo
dist package.
-For other operating systems, the native map can be built from the codebase in
two ways-
-from maven or from the Makefile.</p>
+<p>32-bit and 64-bit Linux and Mac OS X versions of the native map can be
built from the
+Accumulo bin package by executing
<code>$ACCUMULO_HOME/bin/build_native_library.sh</code>.
+If your system's default compiler options are insufficient, you can add
additional compiler
+options to the command line, such as options for the architecture. These will
be passed
+to the Makefile in the environment variable USERFLAGS.</p>
+<p>Examples:</p>
</div>
<div class="olist arabic">
<ol class="arabic">
<li>
-<p>Build from maven using the following command: <code>mvn clean package
-Pnative</code>.</p>
+<p><code>$ACCUMULO_HOME/bin/build_native_library.sh</code></p>
</li>
<li>
-<p>Build from the c++ source by running <code>make</code> in the
<code>$ACCUMULO_HOME/server/src/main/c++</code> directory.</p>
+<p><code>$ACCUMULO_HOME/bin/build_native_library.sh -m32</code></p>
</li>
</ol>
</div>
@@ -3944,7 +3975,9 @@ from maven or from the Makefile.</p>
<p>After building the native map from the source, you will find the artifact in
<code>$ACCUMULO_HOME/lib/native</code>. Upon starting up, the tablet server
will look
in this directory for the map library. If the file is renamed or moved from its
-target directory, the tablet server may not be able to find it.</p>
+target directory, the tablet server may not be able to find it. The system can
also
+locate the native maps shared library by setting <code>LD_LIBRARY_PATH</code>
(or
+<code>DYLD_LIBRARY_PATH</code> on Mac OS X) in
<code>$ACCUMULO_HOME/conf/accumulo-env.sh</code></p>
</div>
</div>
<div class="sect3">
@@ -3997,7 +4030,9 @@ settings between processes and helps fin
</div>
<div class="paragraph">
<p>The instance needs a secret to enable secure communication between servers.
Configure your
-secret and make sure that the <code>accumulo-site.xml</code> file is not
readable to other users.</p>
+secret and make sure that the <code>accumulo-site.xml</code> file is not
readable to other users.
+For alternatives to storing the <code>instance.secret</code> in plaintext,
please read the
+<code>Sensitive Configuration Values</code> section.</p>
</div>
<div class="paragraph">
<p>Some settings can be modified via the Accumulo shell and take effect
immediately, but
@@ -4013,6 +4048,69 @@ from the <code>$ACCUMULO_HOME/conf/</cod
specified in the slaves file.</p>
</div>
</div>
+<div class="sect3">
+<h4 id="_sensitive_configuration_values">11.5.6. Sensitive Configuration
Values</h4>
+<div class="paragraph">
+<p>Accumulo has a number of properties that can be specified via the
accumulo-site.xml
+file which are sensitive in nature, instance.secret and
trace.token.property.password
+are two common examples. Both of these properties, if compromised, have the
ability
+to result in data being leaked to users who should not have access to that
data.</p>
+</div>
+<div class="paragraph">
+<p>In Hadoop-2.6.0, a new CredentialProvider class was introduced which serves
as a common
+implementation to abstract away the storage and retrieval of passwords from
plaintext
+storage in configuration files. Any Property marked with the
<code>Sensitive</code> annotation
+is a candidate for use with these CredentialProviders. For version of Hadoop
which lack
+these classes, the feature will just be unavailable for use.</p>
+</div>
+<div class="paragraph">
+<p>A comma separated list of CredentialProviders can be configured using the
Accumulo Property
+<code>general.security.credential.provider.paths</code>. Each configured URL
will be consulted
+when the Configuration object for accumulo-site.xml is accessed.</p>
+</div>
+</div>
+<div class="sect3">
+<h4 id="_using_a_javakeystorecredentialprovider_for_storage">11.5.7. Using a
JavaKeyStoreCredentialProvider for storage</h4>
+<div class="paragraph">
+<p>One of the implementations provided in Hadoop-2.6.0 is a Java KeyStore
CredentialProvider.
+Each entry in the KeyStore is the Accumulo Property key name. For example, to
store the
+\texttt{instance.secret}, the following command can be used:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>hadoop credential create instance.secret --provider
jceks://file/etc/accumulo/conf/accumulo.jceks</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>The command will then prompt you to enter the secret to use and create a
keystore in:</p>
+</div>
+<div class="literalblock">
+<div class="content">
+<pre>/etc/accumulo/conf/accumulo.jceks</pre>
+</div>
+</div>
+<div class="paragraph">
+<p>Then, accumulo-site.xml must be configured to use this KeyStore as a
CredentialProvider:</p>
+</div>
+<div class="listingblock">
+<div class="content">
+<pre class="highlight"><code class="xml language-xml"><property>
+ <name>general.security.credential.provider.paths</name>
+ <value>jceks://file/etc/accumulo/conf/accumulo.jceks</value>
+</property></code></pre>
+</div>
+</div>
+<div class="paragraph">
+<p>This configuration will then transparently extract the
<code>instance.secret</code> from
+the configured KeyStore and alleviates a human readable storage of the
sensitive
+property.</p>
+</div>
+<div class="paragraph">
+<p>A KeyStore can also be stored in HDFS, which will make the KeyStore readily
available to
+all Accumulo servers. If the local filesystem is used, be aware that each
Accumulo server
+will expect the KeyStore in the same location.</p>
+</div>
+</div>
</div>
<div class="sect2">
<h3 id="_initialization">11.6. Initialization</h3>
@@ -5344,7 +5442,7 @@ the gap between how many splits the old
<div class="paragraph">
<p>For each table, make a new directory that you can move (or copy if you have
the HDFS space to do so)
all of the rfiles for a given table into. For example, to process the table
with an ID of <code>1</code>, make a new directory,
-say <code>/new-table-1</code> and then copy all files from
<code>/accumulo/tables/1/<strong>/</strong>.rf</code> into that directory.
Additionally,
+say <code>/new-table-1</code> and then copy all files from
<code>/accumulo/tables/1/*/*.rf</code> into that directory. Additionally,
make a directory, <code>/new-table-1-failures</code>, for any failures during
the import process. Then, issue the import
command using the Accumulo shell into the new table, telling Accumulo to not
re-set the timestamp:</p>
</div>
@@ -5380,7 +5478,25 @@ omission of new data.</p>
</div>
</div>
<div class="sect2">
-<h3 id="_file_naming_conventions">13.9. File Naming Conventions</h3>
+<h3 id="_upgrade_issues">13.9. Upgrade Issues</h3>
+<div class="paragraph">
+<p><strong>Q</strong>: I upgrade from 1.4 to 1.5 to 1.6 but still have some
WAL files on local disk. Do I have any way
+to recover them?</p>
+</div>
+<div class="paragraph">
+<p><strong>A</strong>: Yes, you can recover them by running the
LocalWALRecovery utility on each node that needs
+recovery performed. The utility will default to using the directory specified
by <code>logger.dir.walog</code> in your
+configuration, or can be overriden by using the
<code>--local-wal-directories</code> option on the tool. It can be
+invoked as follows:</p>
+<div class="literalblock">
+<div class="content">
+<pre>$ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.tserver.log.LocalWALRecovery</pre>
+</div>
+</div>
+</div>
+</div>
+<div class="sect2">
+<h3 id="_file_naming_conventions">13.10. File Naming Conventions</h3>
<div class="paragraph">
<p><strong>Q</strong>: Why are files named like they are? Why do some start
with <code>C</code> and others with <code>F</code>?</p>
</div>
