http://git-wip-us.apache.org/repos/asf/accumulo/blob/e99ec9f0/docs/src/main/asciidoc/chapters/kerberos.txt ---------------------------------------------------------------------- diff --git a/docs/src/main/asciidoc/chapters/kerberos.txt b/docs/src/main/asciidoc/chapters/kerberos.txt deleted file mode 100644 index 743285a..0000000 --- a/docs/src/main/asciidoc/chapters/kerberos.txt +++ /dev/null @@ -1,663 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one or more -// contributor license agreements. See the NOTICE file distributed with -// this work for additional information regarding copyright ownership. -// The ASF licenses this file to You under the Apache License, Version 2.0 -// (the "License"); you may not use this file except in compliance with -// the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -== Kerberos - -=== Overview - -Kerberos is a network authentication protocol that provides a secure way for -peers to prove their identity over an unsecure network in a client-server model. -A centralized key-distribution center (KDC) is the service that coordinates -authentication between a client and a server. Clients and servers use "tickets", -obtained from the KDC via a password or a special file called a "keytab", to -communicate with the KDC and prove their identity. A KDC administrator must -create the principal (name for the client/server identiy) and the password -or keytab, securely passing the necessary information to the actual user/service. -Properly securing the KDC and generated ticket material is central to the security -model and is mentioned only as a warning to administrators running their own KDC. - -To interact with Kerberos programmatically, GSSAPI and SASL are two standards -which allow cross-language integration with Kerberos for authentication. GSSAPI, -the generic security service application program interface, is a standard which -Kerberos implements. In the Java programming language, the language itself also implements -GSSAPI which is leveraged by other applications, like Apache Hadoop and Apache Thrift. -SASL, simple authentication and security layer, is a framework for authentication and -and security over the network. SASL provides a number of mechanisms for authentication, -one of which is GSSAPI. Thus, SASL provides the transport which authenticates -using GSSAPI that Kerberos implements. - -Kerberos is a very complicated software application and is deserving of much -more description than can be provided here. An http://www.roguelynn.com/words/explain-like-im-5-kerberos/[explain like -I'm 5] blog post is very good at distilling the basics, while http://web.mit.edu/kerberos/[MIT Kerberos's project page] -contains lots of documentation for users or administrators. Various Hadoop "vendors" -also provide free documentation that includes step-by-step instructions for -configuring Hadoop and ZooKeeper (which will be henceforth considered as prerequisites). - -=== Within Hadoop - -Out of the box, HDFS and YARN have no ability to enforce that a user is who -they claim they are. Thus, any basic Hadoop installation should be treated as -unsecure: any user with access to the cluster has the ability to access any data. -Using Kerberos to provide authentication, users can be strongly identified, delegating -to Kerberos to determine who a user is and enforce that a user is who they claim to be. -As such, Kerberos is widely used across the entire Hadoop ecosystem for strong -authentication. Since server processes accessing HDFS or YARN are required -to use Kerberos to authenticate with HDFS, it makes sense that they also require -Kerberos authentication from their clients, in addition to other features provided -by SASL. - -A typical deployment involves the creation of Kerberos principals for all server -processes (Hadoop datanodes and namenode(s), ZooKeepers), the creation of a keytab -file for each principal and then proper configuration for the Hadoop site xml files. -Users also need Kerberos principals created for them; however, a user typically -uses a password to identify themselves instead of a keytab. Users can obtain a -ticket granting ticket (TGT) from the KDC using their password which allows them -to authenticate for the lifetime of the TGT (typically one day by default) and alleviates -the need for further password authentication. - -For client server applications, like web servers, a keytab can be created which -allow for fully-automated Kerberos identification removing the need to enter any -password, at the cost of needing to protect the keytab file. These principals -will apply directly to authentication for clients accessing Accumulo and the -Accumulo processes accessing HDFS. - -=== Delegation Tokens - -MapReduce, a common way that clients interact with Accumulo, does not map well to the -client-server model that Kerberos was originally designed to support. Specifically, the parallelization -of tasks across many nodes introduces the problem of securely sharing the user credentials across -these tasks in as safe a manner as possible. To address this problem, Hadoop introduced the notion -of a delegation token to be used in distributed execution settings. - -A delegation token is nothing more than a short-term, on-the-fly password generated after authenticating with the user's -credentials. In Hadoop itself, the Namenode and ResourceManager, for HDFS and YARN respectively, act as the gateway for -delegation tokens requests. For example, before a YARN job is submitted, the implementation will request delegation -tokens from the NameNode and ResourceManager so the YARN tasks can communicate with HDFS and YARN. In the same manner, -support has been added in the Accumulo Master to generate delegation tokens to enable interaction with Accumulo via -MapReduce when Kerberos authentication is enabled in a manner similar to HDFS and YARN. - -Generating an expiring password is, arguably, more secure than distributing the user's -credentials across the cluster as only access to HDFS, YARN and Accumulo would be -compromised in the case of the token being compromised as opposed to the entire -Kerberos credential. Additional details for clients and servers will be covered -in subsequent sections. - -=== Configuring Accumulo - -To configure Accumulo for use with Kerberos, both client-facing and server-facing -changes must be made for a functional system on secured Hadoop. As previously mentioned, -numerous guidelines already exist on the subject of configuring Hadoop and ZooKeeper for -use with Kerberos and won't be covered here. It is assumed that you have functional -Hadoop and ZooKeeper already installed. - -Note that on an existing cluster the server side changes will require a full cluster shutdown and restart. You should -wait to restart the TraceServers until after you've completed the rest of the cluster set up and provisioned -a trace user with appropriate permissions. - -==== Servers - -The first step is to obtain a Kerberos identity for the Accumulo server processes. -When running Accumulo with Kerberos enabled, a valid Kerberos identity will be required -to initiate any RPC between Accumulo processes (e.g. Master and TabletServer) in addition -to any HDFS action (e.g. client to HDFS or TabletServer to HDFS). - -===== Generate Principal and Keytab - -In the +kadmin.local+ shell or using the +-q+ option on +kadmin.local+, create a -principal for Accumulo for all hosts that are running Accumulo processes. A Kerberos -principal is of the form "primary/instance@REALM". "accumulo" is commonly the "primary" -(although not required) and the "instance" is the fully-qualified domain name for -the host that will be running the Accumulo process -- this is required. - ----- -kadmin.local -q "addprinc -randkey accumulo/host.domain.com" ----- - -Perform the above for each node running Accumulo processes in the instance, modifying -"host.domain.com" for your network. The +randkey+ option generates a random password -because we will use a keytab for authentication, not a password, since the Accumulo -server processes don't have an interactive console to enter a password into. - ----- -kadmin.local -q "xst -k accumulo.hostname.keytab accumulo/host.domain.com" ----- - -To simplify deployments, at thet cost of security, all Accumulo principals could -be globbed into a single keytab - ----- -kadmin.local -q "xst -k accumulo.service.keytab -glob accumulo*" ----- - -To ensure that the SASL handshake can occur from clients to servers and servers to servers, -all Accumulo servers must share the same instance and realm principal components as the -"client" needs to know these to set up the connection with the "server". - -===== Server Configuration - -A number of properties need to be changed to account to properly configure servers -in +accumulo-site.xml+. - -[options="header"] -|================================================================= -|Key | Default Value | Description -|general.kerberos.keytab |/etc/security/keytabs/accumulo.service.keytab | -The path to the keytab for Accumulo on local filesystem. Change the value to the actual path on your system. - -|general.kerberos.principal |accumulo/_HOST@REALM | -The Kerberos principal for Accumulo, needs to match the keytab. "_HOST" can be used instead of the actual hostname in the principal and will be automatically expanded to the current FQDN which reduces the configuration file burden. - -|instance.rpc.sasl.enabled |true | -Enables SASL for the Thrift Servers (supports GSSAPI) - -|rpc.sasl.qop |auth | -One of "auth", "auth-int", or "auth-conf". These map to the SASL defined properties for -quality of protection. "auth" is authentication only. "auth-int" is authentication and data -integrity. "auth-conf" is authentication, data integrity and confidentiality. - - -|instance.security.authenticator | -org.apache.accumulo.server.security. -handler.KerberosAuthenticator | -Configures Accumulo to use the Kerberos principal as the Accumulo username/principal - -|instance.security.authorizor | -org.apache.accumulo.server.security. -handler.KerberosAuthorizor | -Configures Accumulo to use the Kerberos principal for authorization purposes - -|instance.security.permissionHandler | -org.apache.accumulo.server.security. -handler.KerberosPermissionHandler| -Configures Accumulo to use the Kerberos principal for permission purposes - -|trace.token.type | -org.apache.accumulo.core.client. -security.tokens.KerberosToken | -Configures the Accumulo Tracer to use the KerberosToken for authentication when serializing traces to the trace table. - -|trace.user |accumulo/_HOST@REALM | -The tracer process needs valid credentials to serialize traces to Accumulo. While the other server processes are -creating a SystemToken from the provided keytab and principal, we can still use a normal KerberosToken and the same -keytab/principal to serialize traces. Like non-Kerberized instances, the table must be created and permissions granted -to the trace.user. The same +_HOST+ replacement is performed on this value, substituted the FQDN for +_HOST+. - -|trace.token.property.keytab || -You can optionally specify the path to a keytab file for the principal given in the +trace.user+ property. If you don't -set this path, it will default to the value given in +general.kerberos.principal+. - -|general.delegation.token.lifetime |7d | -The length of time that the server-side secret used to create delegation tokens is valid. After a server-side secret -expires, a delegation token created with that secret is no longer valid. - -|general.delegation.token.update.interval|1d | -The frequency in which new server-side secrets should be generated to create delegation tokens for clients. Generating -new secrets reduces the likelihood of cryptographic attacks. - -|================================================================= - -Although it should be a prerequisite, it is ever important that you have DNS properly -configured for your nodes and that Accumulo is configured to use the FQDN. It -is extremely important to use the FQDN in each of the "hosts" files for each -Accumulo process: +masters+, +monitors+, +tservers+, +tracers+, and +gc+. - -Normally, no changes are needed in +accumulo-env.sh+ to enable Kerberos. Typically, the +krb5.conf+ -is installed on the local machine in +/etc/+, and the Java library implementations will look -here to find the necessary configuration to communicate with the KDC. Some installations -may require a different +krb5.conf+ to be used for Accumulo which can be accomplished -by adding the JVM system property +-Djava.security.krb5.conf=/path/to/other/krb5.conf+ to -+JAVA_OPTS+ in +accumulo-env.sh+. - -===== KerberosAuthenticator - -The +KerberosAuthenticator+ is an implementation of the pluggable security interfaces -that Accumulo provides. It builds on top of what the default ZooKeeper-based implementation, -but removes the need to create user accounts with passwords in Accumulo for clients. As -long as a client has a valid Kerberos identity, they can connect to and interact with -Accumulo, but without any permissions (e.g. cannot create tables or write data). Leveraging -ZooKeeper removes the need to change the permission handler and authorizor, so other Accumulo -functions regarding permissions and cell-level authorizations do not change. - -It is extremely important to note that, while user operations like +SecurityOperations.listLocalUsers()+, -+SecurityOperations.dropLocalUser()+, and +SecurityOperations.createLocalUser()+ will not return -errors, these methods are not equivalent to normal installations, as they will only operate on -users which have, at one point in time, authenticated with Accumulo using their Kerberos identity. -The KDC is still the authoritative entity for user management. The previously mentioned methods -are provided as they simplify management of users within Accumulo, especially with respect -to granting Authorizations and Permissions to new users. - -===== Administrative User - -Out of the box (without Kerberos enabled), Accumulo has a single user with administrative permissions "root". -This users is used to "bootstrap" other users, creating less-privileged users for applications using -the system. In Kerberos, to authenticate with the system, it's required that the client presents Kerberos -credentials for the principal (user) the client is trying to authenticate as. - -Because of this, an administrative user named "root" would be useless in an instance using Kerberos, -because it is very unlikely to have Kerberos credentials for a principal named `root`. When Kerberos is -enabled, Accumulo will prompt for the name of a user to grant the same permissions as what the `root` -user would normally have. The name of the Accumulo user to grant administrative permissions to can -also be given by the `-u` or `--user` options. - -If you are enabling Kerberos on an existing cluster, you will need to reinitialize the security system in -order to replace the existing "root" user with one that can be used with Kerberos. These steps should be -completed after you have done the previously described configuration changes and will require access to -a complete +accumulo-site.xml+, including the instance secret. Note that this process will delete all -existing users in the system; you will need to reassign user permissions based on Kerberos principals. - -1. Ensure Accumulo is not running. -2. Given the path to a +accumulo-site.xml+ with the instance secret, run the security reset tool. If you are -prompted for a password you can just hit return, since it won't be used. -3. Start the Accumulo cluster - ----- -$ accumulo-cluster stop -... -$ accumulo init --reset-security -Running against secured HDFS -Principal (user) to grant administrative privileges to : acculumo_ad...@example.com -Enter initial password for accumulo_ad...@example.com (this may not be applicable for your security setup): -Confirm initial password for accumulo_ad...@example.com: -$ accumulo-cluster start -... -$ ----- - -===== Verifying secure access - -To verify that servers have correctly started with Kerberos enabled, ensure that the processes -are actually running (they should exit immediately if login fails) and verify that you see -something similar to the following in the application log. - ----- -2015-01-07 11:57:56,826 [security.SecurityUtil] INFO : Attempting to login with keytab as accumulo/hostn...@example.com -2015-01-07 11:57:56,830 [security.UserGroupInformation] INFO : Login successful for user accumulo/hostn...@example.com using keytab file /etc/security/keytabs/accumulo.service.keytab ----- - -===== Impersonation - -Impersonation is functionality which allows a certain user to act as another. One direct application -of this concept within Accumulo is the Thrift proxy. The Thrift proxy is configured to accept -user requests and pass them onto Accumulo, enabling client access to Accumulo via any thrift-compatible -language. When the proxy is running with SASL transports, this enforces that clients present a valid -Kerberos identity to make a connection. In this situation, the Thrift proxy server does not have -access to the secret key material in order to make a secure connection to Accumulo as the client, -it can only connect to Accumulo as itself. Impersonation, in this context, refers to the ability -of the proxy to authenticate to Accumulo as itself, but act on behalf of an Accumulo user. - -Accumulo supports basic impersonation of end-users by a third party via static rules in Accumulo's -site configuration file. These two properties are semi-colon separated properties which are aligned -by index. This first element in the user impersonation property value matches the first element -in the host impersonation property value, etc. - ----- -<property> - <name>instance.rpc.sasl.allowed.user.impersonation</name> - <value>$PROXY_USER:*</value> -</property> - -<property> - <name>instance.rpc.sasl.allowed.host.impersonation</name> - <value>*</value> -</property> ----- - -Here, +$PROXY_USER+ can impersonate any user from any host. - -The following is an example of specifying a subset of users +$PROXY_USER+ can impersonate and also -limiting the hosts from which +$PROXY_USER+ can initiate requests from. - ----- -<property> - <name>instance.rpc.sasl.allowed.user.impersonation</name> - <value>$PROXY_USER:user1,user2;$PROXY_USER2:user2,user4</value> -</property> - -<property> - <name>instance.rpc.sasl.allowed.host.impersonation</name> - <value>host1.domain.com,host2.domain.com;*</value> -</property> ----- - -Here, +$PROXY_USER+ can impersonate user1 and user2 only from host1.domain.com or host2.domain.com. -+$PROXY_USER2+ can impersonate user2 and user4 from any host. - -In these examples, the value +$PROXY_USER+ is the Kerberos principal of the server which is acting on behalf of a user. -Impersonation is enforced by the Kerberos principal and the host from which the RPC originated (from the perspective -of the Accumulo TabletServers/Masters). An asterisk (*) can be used to specify all users or all hosts (depending on the context). - -===== Delegation Tokens - -Within Accumulo services, the primary task to implement delegation tokens is the generation and distribution -of a shared secret among all Accumulo tabletservers and the master. The secret key allows for generation -of delegation tokens for users and verification of delegation tokens presented by clients. If a server -process is unaware of the secret key used to create a delegation token, the client cannot be authenticated. -As ZooKeeper distribution is an asynchronous operation (typically on the order of seconds), the -value for `general.delegation.token.update.interval` should be on the order of hours to days to reduce the -likelihood of servers rejecting valid clients because the server did not yet see a new secret key. - -Supporting authentication with both Kerberos credentials and delegation tokens, the SASL thrift -server accepts connections with either `GSSAPI` and `DIGEST-MD5` mechanisms set. The `DIGEST-MD5` mechanism -enables authentication as a normal username and password exchange which `DelegationToken`s leverages. - -Since delegation tokens are a weaker form of authentication than Kerberos credentials, user access -to obtain delegation tokens from Accumulo is protected with the `DELEGATION_TOKEN` system permission. Only -users with the system permission are allowed to obtain delegation tokens. It is also recommended -to configure confidentiality with SASL, using the `rpc.sasl.qop=auth-conf` configuration property, to -ensure that prying eyes cannot view the `DelegationToken` as it passes over the network. - ----- -# Check a user's permissions -admin@REALM@accumulo> userpermissions -u user@REALM - -# Grant the DELEGATION_TOKEN system permission to a user -admin@REALM@accumulo> grant System.DELEGATION_TOKEN -s -u user@REALM ----- - -==== Clients - -===== Create client principal - -Like the Accumulo servers, clients must also have a Kerberos principal created for them. The -primary difference between a server principal is that principals for users are created -with a password and also not qualified to a specific instance (host). - ----- -kadmin.local -q "addprinc $user" ----- - -The above will prompt for a password for that user which will be used to identify that $user. -The user can verify that they can authenticate with the KDC using the command `kinit $user`. -Upon entering the correct password, a local credentials cache will be made which can be used -to authenticate with Accumulo, access HDFS, etc. - -The user can verify the state of their local credentials cache by using the command `klist`. - ----- -$ klist -Ticket cache: FILE:/tmp/krb5cc_123 -Default principal: u...@example.com - -Valid starting Expires Service principal -01/07/2015 11:56:35 01/08/2015 11:56:35 krbtgt/example....@example.com - renew until 01/14/2015 11:56:35 ----- - -===== Configuration - -The second thing clients need to do is to set up their client configuration file. By -default, this file is stored in +~/.accumulo/config+ or +/path/to/accumulo/client.conf+. -Accumulo utilities also allow you to provide your own copy of this file in any location -using the +--config-file+ command line option. - -Three items need to be set to enable access to Accumulo: - -* +instance.rpc.sasl.enabled+=_true_ -* +rpc.sasl.qop+=_auth_ -* +kerberos.server.primary+=_accumulo_ - -Each of these properties *must* match the configuration of the accumulo servers; this is -required to set up the SASL transport. - -===== Verifying Administrative Access - -At this point you should have enough configured on the server and client side to interact with -the system. You should verify that the administrative user you chose earlier can successfully -interact with the sytem. - -While this example logs in via +kinit+ with a password, any login method that caches Kerberos tickets -should work. - ----- -$ kinit accumulo_ad...@example.com -Password for accumulo_ad...@example.com: ****************************** -$ accumulo shell - -Shell - Apache Accumulo Interactive Shell -- -- version: 1.7.2 -- instance name: MYACCUMULO -- instance id: 483b9038-889f-4b2d-b72b-dfa2bb5dbd07 -- -- type 'help' for a list of available commands -- -accumulo_ad...@example.com@MYACCUMULO> userpermissions -System permissions: System.GRANT, System.CREATE_TABLE, System.DROP_TABLE, System.ALTER_TABLE, System.CREATE_USER, System.DROP_USER, System.ALTER_USER, System.SYSTEM, System.CREATE_NAMESPACE, System.DROP_NAMESPACE, System.ALTER_NAMESPACE, System.OBTAIN_DELEGATION_TOKEN - -Namespace permissions (accumulo): Namespace.READ, Namespace.ALTER_TABLE - -Table permissions (accumulo.metadata): Table.READ, Table.ALTER_TABLE -Table permissions (accumulo.replication): Table.READ -Table permissions (accumulo.root): Table.READ, Table.ALTER_TABLE - -accumulo_ad...@example.com@MYACCUMULO> quit -$ kdestroy -$ ----- - -===== DelegationTokens with MapReduce - -To use DelegationTokens in a custom MapReduce job, the call to `setConnectorInfo()` method -on `AccumuloInputFormat` or `AccumuloOutputFormat` should be the only necessary change. Instead -of providing an instance of a `KerberosToken`, the user must call `SecurityOperations.getDelegationToken` -using a `Connector` obtained with that `KerberosToken`, and pass the `DelegationToken` to -`setConnectorInfo` instead of the `KerberosToken`. It is expected that the user launching -the MapReduce job is already logged in via Kerberos via a keytab or via a locally-cached -Kerberos ticket-granting-ticket (TGT). - -[source,java] ----- -Instance instance = getInstance(); -KerberosToken kt = new KerberosToken(); -Connector conn = instance.getConnector(principal, kt); -DelegationToken dt = conn.securityOperations().getDelegationToken(); - -// Reading from Accumulo -AccumuloInputFormat.setConnectorInfo(job, principal, dt); - -// Writing to Accumulo -AccumuloOutputFormat.setConnectorInfo(job, principal, dt); ----- - -If the user passes a `KerberosToken` to the `setConnectorInfo` method, the implementation will -attempt to obtain a `DelegationToken` automatically, but this does have limitations -based on the other MapReduce configuration methods already called and permissions granted -to the calling user. It is best for the user to acquire the DelegationToken on their own -and provide it directly to `setConnectorInfo`. - -Users must have the `DELEGATION_TOKEN` system permission to call the `getDelegationToken` -method. The obtained delegation token is only valid for the requesting user for a period -of time dependent on Accumulo's configuration (`general.delegation.token.lifetime`). - -It is also possible to obtain and use `DelegationToken`s outside of the context -of MapReduce. - -[source,java] ----- -String principal = "user@REALM"; -Instance instance = getInstance(); -Connector connector = instance.getConnector(principal, new KerberosToken()); -DelegationToken delegationToken = connector.securityOperations().getDelegationToken(); - -Connector dtConnector = instance.getConnector(principal, delegationToken); ----- - -Use of the `dtConnector` will perform each operation as the original user, but without -their Kerberos credentials. - -For the duration of validity of the `DelegationToken`, the user *must* take the necessary precautions -to protect the `DelegationToken` from prying eyes as it can be used by any user on any host to impersonate -the user who requested the `DelegationToken`. YARN ensures that passing the delegation token from the client -JVM to each YARN task is secure, even in multi-tenant instances. - -==== Debugging - -*Q*: I have valid Kerberos credentials and a correct client configuration file but -I still get errors like: - ----- -java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ----- - -*A*: When you have a valid client configuration and Kerberos TGT, it is possible that the search -path for your local credentials cache is incorrect. Check the value of the KRB5CCNAME environment -value, and ensure it matches the value reported by `klist`. - ----- -$ echo $KRB5CCNAME - -$ klist -Ticket cache: FILE:/tmp/krb5cc_123 -Default principal: u...@example.com - -Valid starting Expires Service principal -01/07/2015 11:56:35 01/08/2015 11:56:35 krbtgt/example....@example.com - renew until 01/14/2015 11:56:35 -$ export KRB5CCNAME=/tmp/krb5cc_123 -$ echo $KRB5CCNAME -/tmp/krb5cc_123 ----- - -*Q*: I thought I had everything configured correctly, but my client/server still fails to log in. -I don't know what is actually failing. - -*A*: Add the following system property to the JVM invocation: - ----- --Dsun.security.krb5.debug=true ----- - -This will enable lots of extra debugging at the JVM level which is often sufficient to -diagnose some high-level configuration problem. Client applications can add this system property by -hand to the command line and Accumulo server processes or applications started using the `accumulo` -script by adding the property to +JAVA_OPTS+ in +accumulo-env.sh+. - -Additionally, you can increase the log4j levels on +org.apache.hadoop.security+, which includes the -Hadoop +UserGroupInformation+ class, which will include some high-level debug statements. This -can be controlled in your client application, or using +log4j-service.properties+ - -*Q*: All of my Accumulo processes successfully start and log in with their -keytab, but they are unable to communicate with each other, showing the -following errors: - ----- -2015-01-12 14:47:27,055 [transport.TSaslTransport] ERROR: SASL negotiation failure -javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)] - at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) - at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) - at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) - at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) - at org.apache.accumulo.core.rpc.UGIAssumingTransport$1.run(UGIAssumingTransport.java:53) - at org.apache.accumulo.core.rpc.UGIAssumingTransport$1.run(UGIAssumingTransport.java:49) - at java.security.AccessController.doPrivileged(Native Method) - at javax.security.auth.Subject.doAs(Subject.java:415) - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) - at org.apache.accumulo.core.rpc.UGIAssumingTransport.open(UGIAssumingTransport.java:49) - at org.apache.accumulo.core.rpc.ThriftUtil.createClientTransport(ThriftUtil.java:357) - at org.apache.accumulo.core.rpc.ThriftUtil.createTransport(ThriftUtil.java:255) - at org.apache.accumulo.server.master.LiveTServerSet$TServerConnection.getTableMap(LiveTServerSet.java:106) - at org.apache.accumulo.master.Master.gatherTableInformation(Master.java:996) - at org.apache.accumulo.master.Master.access$600(Master.java:160) - at org.apache.accumulo.master.Master$StatusThread.updateStatus(Master.java:911) - at org.apache.accumulo.master.Master$StatusThread.run(Master.java:901) -Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER) - at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710) - at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) - at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) - at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) - ... 16 more -Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER - at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) - at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192) - at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203) - at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:309) - at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115) - at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:454) - at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641) - ... 19 more -Caused by: KrbException: Identifier doesn't match expected value (906) - at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143) - at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66) - at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61) - at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) - ... 25 more ----- - -or - ----- -2015-01-12 14:47:29,440 [server.TThreadPoolServer] ERROR: Error occurred during processing of message. -java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed - at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) - at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51) - at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48) - at java.security.AccessController.doPrivileged(Native Method) - at javax.security.auth.Subject.doAs(Subject.java:356) - at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1608) - at org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48) - at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208) - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) - at java.lang.Thread.run(Thread.java:745) -Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: GSS initiate failed - at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:190) - at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125) - at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) - at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) - at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) - ... 10 more ----- - -*A*: As previously mentioned, the hostname, and subsequently the address each Accumulo process is bound/listening -on, is extremely important when negotiating an SASL connection. This problem commonly arises when the Accumulo -servers are not configured to listen on the address denoted by their FQDN. - -The values in the Accumulo "hosts" files (In +accumulo/conf+: +masters+, +monitors+, +tservers+, +tracers+, -and +gc+) should match the instance componentof the Kerberos server principal (e.g. +host+ in +accumulo/h...@example.com+). - -*Q*: After configuring my system for Kerberos, server processes come up normally and I can interact with the system. However, -when I attempt to use the "Recent Traces" page on the Monitor UI I get a stacktrace similar to: - ----- - java.lang.AssertionError: AuthenticationToken should not be null - at org.apache.accumulo.monitor.servlets.trace.Basic.getScanner(Basic.java:139) - at org.apache.accumulo.monitor.servlets.trace.Summary.pageBody(Summary.java:164) - at org.apache.accumulo.monitor.servlets.BasicServlet.doGet(BasicServlet.java:63) - at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) - at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) - at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) - at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551) - at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) - at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568) - at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) - at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) - at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) - at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) - at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) - at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) - at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) - at org.eclipse.jetty.server.Server.handle(Server.java:462) - at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) - at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) - at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) - at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) - at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) - at java.lang.Thread.run(Thread.java:745) - ----- - -*A*: This indicates that the Monitor has not been able to successfully log in a client-side user to read from the +trace+ table. Accumulo allows the TraceServer to rely on the property +general.kerberos.keytab+ as a fallback when logging in the trace user if the +trace.token.property.keytab+ property isn't defined. Some earlier versions of Accumulo did not do this same fallback for the Monitor's use of the trace user. The end result is that if you configure +general.kerberos.keytab+ and not +trace.token.property.keytab+ you will end up with a system that properly logs trace information but can't view it. - -Ensure you have set +trace.token.property.keytab+ to point to a keytab for the principal defined in +trace.user+ in the +accumulo-site.xml+ file for the Monitor, since that should work in all versions of Accumulo.
http://git-wip-us.apache.org/repos/asf/accumulo/blob/e99ec9f0/docs/src/main/asciidoc/chapters/multivolume.txt ---------------------------------------------------------------------- diff --git a/docs/src/main/asciidoc/chapters/multivolume.txt b/docs/src/main/asciidoc/chapters/multivolume.txt deleted file mode 100644 index 8921f2d..0000000 --- a/docs/src/main/asciidoc/chapters/multivolume.txt +++ /dev/null @@ -1,82 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one or more -// contributor license agreements. See the NOTICE file distributed with -// this work for additional information regarding copyright ownership. -// The ASF licenses this file to You under the Apache License, Version 2.0 -// (the "License"); you may not use this file except in compliance with -// the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -== Multi-Volume Installations - -This is an advanced configuration setting for very large clusters -under a lot of write pressure. - -The HDFS NameNode holds all of the metadata about the files in -HDFS. For fast performance, all of this information needs to be stored -in memory. A single NameNode with 64G of memory can store the -metadata for tens of millions of files.However, when scaling beyond a -thousand nodes, an active Accumulo system can generate lots of updates -to the file system, especially when data is being ingested. The large -number of write transactions to the NameNode, and the speed of a -single edit log, can become the limiting factor for large scale -Accumulo installations. - -You can see the effect of slow write transactions when the Accumulo -Garbage Collector takes a long time (more than 5 minutes) to delete -the files Accumulo no longer needs. If your Garbage Collector -routinely runs in less than a minute, the NameNode is performing well. - -However, if you do begin to experience slow-down and poor GC -performance, Accumulo can be configured to use multiple NameNode -servers. The configuration ``instance.volumes'' should be set to a -comma-separated list, using full URI references to different NameNode -servers: - -[source,xml] -<property> - <name>instance.volumes</name> - <value>hdfs://ns1:9001,hdfs://ns2:9001</value> -</property> - -The introduction of multiple volume support in 1.6 changed the way Accumulo -stores pointers to files. It now stores fully qualified URI references to -files. Before 1.6, Accumulo stored paths that were relative to a table -directory. After an upgrade these relative paths will still exist and are -resolved using instance.dfs.dir, instance.dfs.uri, and Hadoop configuration in -the same way they were before 1.6. - -If the URI for a namenode changes (e.g. namenode was running on host1 and its -moved to host2), then Accumulo will no longer function. Even if Hadoop and -Accumulo configurations are changed, the fully qualified URIs stored in -Accumulo will still contain the old URI. To handle this Accumulo has the -following configuration property for replacing URI stored in its metadata. The -example configuration below will replace ns1 with nsA and ns2 with nsB in -Accumulo metadata. For this property to take affect, Accumulo will need to be -restarted. - -[source,xml] -<property> - <name>instance.volumes.replacements</name> - <value>hdfs://ns1:9001 hdfs://nsA:9001, hdfs://ns2:9001 hdfs://nsB:9001</value> -</property> - -Using viewfs or HA namenode, introduced in Hadoop 2, offers another option for -managing the fully qualified URIs stored in Accumulo. Viewfs and HA namenode -both introduce a level of indirection in the Hadoop configuration. For -example assume viewfs:///nn1 maps to hdfs://nn1 in the Hadoop configuration. -If viewfs://nn1 is used by Accumulo, then its easy to map viewfs://nn1 to -hdfs://nnA by changing the Hadoop configuration w/o doing anything to Accumulo. -A production system should probably use a HA namenode. Viewfs may be useful on -a test system with a single non HA namenode. - -You may also want to configure your cluster to use Federation, -available in Hadoop 2.0, which allows DataNodes to respond to multiple -NameNode servers, so you do not have to partition your DataNodes by -NameNode. http://git-wip-us.apache.org/repos/asf/accumulo/blob/e99ec9f0/docs/src/main/asciidoc/chapters/replication.txt ---------------------------------------------------------------------- diff --git a/docs/src/main/asciidoc/chapters/replication.txt b/docs/src/main/asciidoc/chapters/replication.txt deleted file mode 100644 index fd8905e..0000000 --- a/docs/src/main/asciidoc/chapters/replication.txt +++ /dev/null @@ -1,399 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one or more -// contributor license agreements. See the NOTICE file distributed with -// this work for additional information regarding copyright ownership. -// The ASF licenses this file to You under the Apache License, Version 2.0 -// (the "License"); you may not use this file except in compliance with -// the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -== Replication - -=== Overview - -Replication is a feature of Accumulo which provides a mechanism to automatically -copy data to other systems, typically for the purpose of disaster recovery, -high availability, or geographic locality. It is best to consider this feature -as a framework for automatic replication instead of the ability to copy data -from to another Accumulo instance as copying to another Accumulo cluster is -only an implementation detail. The local Accumulo cluster is hereby referred -to as the +primary+ while systems being replicated to are known as -+peers+. - -This replication framework makes two Accumulo instances, where one instance -replicates to another, eventually consistent between one another, as opposed -to the strong consistency that each single Accumulo instance still holds. That -is to say, attempts to read data from a table on a peer which has pending replication -from the primary will not wait for that data to be replicated before running the scan. -This is desirable for a number of reasons, the most important is that the replication -framework is not limited by network outages or offline peers, but only by the HDFS -space available on the primary system. - -Replication configurations can be considered as a directed graph which allows cycles. -The systems in which data was replicated from is maintained in each Mutation which -allow each system to determine if a peer has already has the data in which -the system wants to send. - -Data is replicated by using the Write-Ahead logs (WAL) that each TabletServer is -already maintaining. TabletServers records which WALs have data that need to be -replicated to the +accumulo.metadata+ table. The Master uses these records, -combined with the local Accumulo table that the WAL was used with, to create records -in the +replication+ table which track which peers the given WAL should be -replicated to. The Master latter uses these work entries to assign the actual -replication task to a local TabletServer using ZooKeeper. A TabletServer will get -a lock in ZooKeeper for the replication of this file to a peer, and proceed to -replicate to the peer, recording progress in the +replication+ table as -data is successfully replicated on the peer. Later, the Master and Garbage Collector -will remove records from the +accumulo.metadata+ and +replication+ tables -and files from HDFS, respectively, after replication to all peers is complete. - -=== Configuration - -Configuration of Accumulo to replicate data to another system can be categorized -into the following sections. - -==== Site Configuration - -Each system involved in replication (even the primary) needs a name that uniquely -identifies it across all peers in the replication graph. This should be considered -fixed for an instance, and set in +accumulo-site.xml+. - ----- -<property> - <name>replication.name</name> - <value>primary</value> - <description>Unique name for this system used by replication</description> -</property> ----- - -==== Instance Configuration - -For each peer of this system, Accumulo needs to know the name of that peer, -the class used to replicate data to that system and some configuration information -to connect to this remote peer. In the case of Accumulo, this additional data -is the Accumulo instance name and ZooKeeper quorum; however, this varies on the -replication implementation for the peer. - -These can be set in the site configuration to ease deployments; however, as they may -change, it can be useful to set this information using the Accumulo shell. - -To configure a peer with the name +peer1+ which is an Accumulo system with an instance name of +accumulo_peer+ -and a ZooKeeper quorum of +10.0.0.1,10.0.2.1,10.0.3.1+, invoke the following -command in the shell. - ----- -root@accumulo_primary> config -s -replication.peer.peer1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,accumulo_peer,10.0.0.1,10.0.2.1,10.0.3.1 ----- - -Since this is an Accumulo system, we also want to set a username and password -to use when authenticating with this peer. On our peer, we make a special user -which has permission to write to the tables we want to replicate data into, "replication" -with a password of "password". We then need to record this in the primary's configuration. - ----- -root@accumulo_primary> config -s replication.peer.user.peer1=replication -root@accumulo_primary> config -s replication.peer.password.peer1=password ----- - -Alternatively, when configuring replication on Accumulo running Kerberos, a keytab -file per peer can be configured instead of a password. The provided keytabs must be readable -by the unix user running Accumulo. They keytab for a peer can be unique from the -keytab used by Accumulo or any keytabs for other peers. - ----- -accum...@example.com@accumulo_primary> config -s replication.peer.user.peer1=replicat...@example.com -accum...@example.com@accumulo_primary> config -s replication.peer.keytab.peer1=/path/to/replication.keytab ----- - -==== Table Configuration - -Now, we presently have a peer defined, so we just need to configure which tables will -replicate to that peer. We also need to configure an identifier to determine where -this data will be replicated on the peer. Since we're replicating to another Accumulo -cluster, this is a table ID. In this example, we want to enable replication on -+my_table+ and configure our peer +accumulo_peer+ as a target, sending -the data to the table with an ID of +2+ in +accumulo_peer+. - ----- -root@accumulo_primary> config -t my_table -s table.replication=true -root@accumulo_primary> config -t my_table -s table.replication.target.accumulo_peer=2 ----- - -To replicate a single table on the primary to multiple peers, the second command -in the above shell snippet can be issued, for each peer and remote identifier pair. - -=== Monitoring - -Basic information about replication status from a primary can be found on the Accumulo -Monitor server, using the +Replication+ link the sidebar. - -On this page, information is broken down into the following sections: - -1. Files pending replication by peer and target -2. Files queued for replication, with progress made - -=== Work Assignment - -Depending on the schema of a table, different implementations of the WorkAssigner used could -be configured. The implementation is controlled via the property +replication.work.assigner+ -and the full class name for the implementation. This can be configured via the shell or -+accumulo-site.xml+. - ----- -<property> - <name>replication.work.assigner</name> - <value>org.apache.accumulo.master.replication.SequentialWorkAssigner</value> - <description>Implementation used to assign work for replication</description> -</property> ----- - ----- -root@accumulo_primary> config -t my_table -s replication.work.assigner=org.apache.accumulo.master.replication.SequentialWorkAssigner ----- - -Two implementations are provided. By default, the +SequentialWorkAssigner+ is configured for an -instance. The SequentialWorkAssigner ensures that, per peer and each remote identifier, each WAL is -replicated in the order in which they were created. This is sufficient to ensure that updates to a table -will be replayed in the correct order on the peer. This implementation has the downside of only replicating -a single WAL at a time. - -The second implementation, the +UnorderedWorkAssigner+ can be used to overcome the limitation -of only a single WAL being replicated to a target and peer at any time. Depending on the table schema, -it's possible that multiple versions of the same Key with different values are infrequent or nonexistent. -In this case, parallel replication to a peer and target is possible without any downsides. In the case -where this implementation is used were column updates are frequent, it is possible that there will be -an inconsistency between the primary and the peer. - -=== ReplicaSystems - -+ReplicaSystem+ is the interface which allows abstraction of replication of data -to peers of various types. Presently, only an +AccumuloReplicaSystem+ is provided -which will replicate data to another Accumulo instance. A +ReplicaSystem+ implementation -is run inside of the TabletServer process, and can be configured as mentioned in the -+Instance Configuration+ section of this document. Theoretically, an implementation -of this interface could send data to other filesystems, databases, etc. - -==== AccumuloReplicaSystem - -The +AccumuloReplicaSystem+ uses Thrift to communicate with a peer Accumulo instance -and replicate the necessary data. The TabletServer running on the primary will communicate -with the Master on the peer to request the address of a TabletServer on the peer which -this TabletServer will use to replicate the data. - -The TabletServer on the primary will then replicate data in batches of a configurable -size (+replication.max.unit.size+). The TabletServer on the peer will report how many -records were applied back to the primary, which will be used to record how many records -were successfully replicated. The TabletServer on the primary will continue to replicate -data in these batches until no more data can be read from the file. - -=== Other Configuration - -There are a number of configuration values that can be used to control how -the implementation of various components operate. - -[width="75%",cols=">,^2,^2"] -[options="header"] -|==== -|Property | Description | Default -|replication.max.work.queue | Maximum number of files queued for replication at one time | 1000 -|replication.work.assignment.sleep | Time between invocations of the WorkAssigner | 30s -|replication.worker.threads | Size of threadpool used to replicate data to peers | 4 -|replication.receipt.service.port | Thrift service port to listen for replication requests, can use '0' for a random port | 10002 -|replication.work.attempts | Number of attempts to replicate to a peer before aborting the attempt | 10 -|replication.receiver.min.threads | Minimum number of idle threads for handling incoming replication | 1 -|replication.receiver.threadcheck.time | Time between attempting adjustments of thread pool for incoming replications | 30s -|replication.max.unit.size | Maximum amount of data to be replicated in one RPC | 64M -|replication.work.assigner | Work Assigner implementation | org.apache.accumulo.master.replication.SequentialWorkAssigner -|tserver.replication.batchwriter.replayer.memory| Size of BatchWriter cache to use in applying replication requests | 50M -|==== - -=== Example Practical Configuration - -A real-life example is now provided to give concrete application of replication configuration. This -example is a two instance Accumulo system, one primary system and one peer system. They are called -primary and peer, respectively. Each system also have a table of the same name, "my_table". The instance -name for each is also the same (primary and peer), and both have ZooKeeper hosts on a node with a hostname -with that name as well (primary:2181 and peer:2181). - -We want to configure these systems so that "my_table" on "primary" replicates to "my_table" on "peer". - -==== accumulo-site.xml - -We can assign the "unique" name that identifies this Accumulo instance among all others that might participate -in replication together. In this example, we will use the names provided in the description. - -===== Primary - ----- -<property> - <name>replication.name</name> - <value>primary</value> - <description>Defines the unique name</description> -</property> ----- - -===== Peer - ----- -<property> - <name>replication.name</name> - <value>peer</value> -</property> ----- - -==== masters and tservers files - -Be *sure* to use non-local IP addresses. Other nodes need to connect to it and using localhost will likely result in -a local node talking to another local node. - -==== Start both instances - -The rest of the configuration is dynamic and is best configured on the fly (in ZooKeeper) than in accumulo-site.xml. - -==== Peer - -The next series of command are to be run on the peer system. Create a user account for the primary instance called -"peer". The password for this account will need to be saved in the configuration on the primary - ----- -root@peer> createtable my_table -root@peer> createuser peer -root@peer> grant -t my_table -u peer Table.WRITE -root@peer> grant -t my_table -u peer Table.READ -root@peer> tables -l ----- - -Remember what the table ID for 'my_table' is. You'll need that to configured the primary instance. - -==== Primary - -Next, configure the primary instance. - -===== Set up the table - ----- -root@primary> createtable my_table ----- - -===== Define the Peer as a replication peer to the Primary - -We're defining the instance with replication.name of 'peer' as a peer. We provide the implementation of ReplicaSystem -that we want to use, and the configuration for the AccumuloReplicaSystem. In this case, the configuration is the Accumulo -Instance name for 'peer' and the ZooKeeper quorum string. The configuration key is of the form -"replication.peer.$peer_name". - ----- -root@primary> config -s replication.peer.peer=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,peer,$peer_zk_quorum ----- - -===== Set the authentication credentials - -We want to use that special username and password that we created on the peer, so we have a means to write data to -the table that we want to replicate to. The configuration key is of the form "replication.peer.user.$peer_name". - ----- -root@primary> config -s replication.peer.user.peer=peer -root@primary> config -s replication.peer.password.peer=peer ----- - -===== Enable replication on the table - -Now that we have defined the peer on the primary and provided the authentication credentials, we need to configure -our table with the implementation of ReplicaSystem we want to use to replicate to the peer. In this case, our peer -is an Accumulo instance, so we want to use the AccumuloReplicaSystem. - -The configuration for the AccumuloReplicaSystem is the table ID for the table on the peer instance that we -want to replicate into. Be sure to use the correct value for $peer_table_id. The configuration key is of -the form "table.replication.target.$peer_name". - ----- -root@primary> config -t my_table -s table.replication.target.peer=$peer_table_id ----- - -Finally, we can enable replication on this table. - ----- -root@primary> config -t my_table -s table.replication=true ----- - -=== Extra considerations for use - -While this feature is intended for general-purpose use, its implementation does carry some baggage. Like any software, -replication is a feature that operates well within some set of use cases but is not meant to support all use cases. -For the benefit of the users, we can enumerate these cases. - -==== Latency - -As previously mentioned, the replication feature uses the Write-Ahead Log files for a number of reasons, one of which -is to prevent the need for data to be written to RFiles before it is available to be replicated. While this can help -reduce the latency for a batch of Mutations that have been written to Accumulo, the latency is at least seconds to tens -of seconds for replication once ingest is active. For a table which replication has just been enabled on, this is likely -to take a few minutes before replication will begin. - -Once ingest is active and flowing into the system at a regular rate, replication should be occurring at a similar rate, -given sufficient computing resources. Replication attempts to copy data at a rate that is to be considered low latency -but is not a replacement for custom indexing code which can ensure near real-time referential integrity on secondary indexes. - -==== Table-Configured Iterators - -Accumulo Iterators tend to be a heavy hammer which can be used to solve a variety of problems. In general, it is highly -recommended that Iterators which are applied at major compaction time are both idempotent and associative due to the -non-determinism in which some set of files for a Tablet might be compacted. In practice, this translates to common patterns, -such as aggregation, which are implemented in a manner resilient to duplication (such as using a Set instead of a List). - -Due to the asynchronous nature of replication and the expectation that hardware failures and network partitions will exist, -it is generally not recommended to not configure replication on a table which has Iterators set which are not idempotent. -While the replication implementation can make some simple assertions to try to avoid re-replication of data, it is not -presently guaranteed that all data will only be sent to a peer once. Data will be replicated at least once. Typically, -this is not a problem as the VersioningIterator will automaticaly deduplicate this over-replication because they will -have the same timestamp; however, certain Combiners may result in inaccurate aggregations. - -As a concrete example, consider a table which has the SummingCombiner configured to sum all values for -multiple versions of the same Key. For some key, consider a set of numeric values that are written to a table on the -primary: [1, 2, 3]. On the primary, all of these are successfully written and thus the current value for the given key -would be 6, (1 + 2 + 3). Consider, however, that each of these updates to the peer were done independently (because -other data was also included in the write-ahead log that needed to be replicated). The update with a value of 1 was -successfully replicated, and then we attempted to replicate the update with a value of 2 but the remote server never -responded. The primary does not know whether the update with a value of 2 was actually applied or not, so the -only recourse is to re-send the update. After we receive confirmation that the update with a value of 2 was replicated, -we will then replicate the update with 3. If the peer did never apply the first update of '2', the summation is accurate. -If the update was applied but the acknowledgement was lost for some reason (system failure, network partition), the -update will be resent to the peer. Because addition is non-idempotent, we have created an inconsistency between the -primary and peer. As such, the SummingCombiner wouldn't be recommended on a table being replicated. - -While there are changes that could be made to the replication implementation which could attempt to mitigate this risk, -presently, it is not recommended to configure Iterators or Combiners which are not idempotent to support cases where -inaccuracy of aggregations is not acceptable. - -==== Duplicate Keys - -In Accumulo, when more than one key exists that are exactly the same, keys that are equal down to the timestamp, -the retained value is non-deterministic. Replication introduces another level of non-determinism in this case. -For a table that is being replicated and has multiple equal keys with different values inserted into it, the final -value in that table on the primary instance is not guaranteed to be the final value on all replicas. - -For example, say the values that were inserted on the primary instance were +value1+ and +value2+ and the final -value was +value1+, it is not guaranteed that all replicas will have +value1+ like the primary. The final value is -non-deterministic for each instance. - -As is the recommendation without replication enabled, if multiple values for the same key (sans timestamp) are written to -Accumulo, it is strongly recommended that the value in the timestamp properly reflects the intended version by -the client. That is to say, newer values inserted into the table should have larger timestamps. If the time between -writing updates to the same key is significant (order minutes), this concern can likely be ignored. - -==== Bulk Imports - -Currently, files that are bulk imported into a table configured for replication are not replicated. There is no -technical reason why it was not implemented, it was simply omitted from the initial implementation. This is considered a -fair limitation because bulk importing generated files multiple locations is much simpler than bifurcating "live" ingest -data into two instances. Given some existing bulk import process which creates files and them imports them into an -Accumulo instance, it is trivial to copy those files to a new HDFS instance and import them into another Accumulo -instance using the same process. Hadoop's +distcp+ command provides an easy way to copy large amounts of data to another -HDFS instance which makes the problem of duplicating bulk imports very easy to solve. http://git-wip-us.apache.org/repos/asf/accumulo/blob/e99ec9f0/docs/src/main/asciidoc/chapters/sampling.txt ---------------------------------------------------------------------- diff --git a/docs/src/main/asciidoc/chapters/sampling.txt b/docs/src/main/asciidoc/chapters/sampling.txt deleted file mode 100644 index 237f43c..0000000 --- a/docs/src/main/asciidoc/chapters/sampling.txt +++ /dev/null @@ -1,86 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one or more -// contributor license agreements. See the NOTICE file distributed with -// this work for additional information regarding copyright ownership. -// The ASF licenses this file to You under the Apache License, Version 2.0 -// (the "License"); you may not use this file except in compliance with -// the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -== Sampling - -=== Overview - -Accumulo has the ability to generate and scan a per table set of sample data. -This sample data is kept up to date as a table is mutated. What key values are -placed in the sample data is configurable per table. - -This feature can be used for query estimation and optimization. For an example -of estimation assume an Accumulo table is configured to generate a sample -containing one millionth of a tables data. If a query is executed against the -sample and returns one thousand results, then the same query against all the -data would probably return a billion results. A nice property of having -Accumulo generate the sample is that its always up to date. So estimations -will be accurate even when querying the most recently written data. - -An example of a query optimization is an iterator using sample data to get an -estimate, and then making decisions based on the estimate. - -=== Configuring - -Inorder to use sampling, an Accumulo table must be configured with a class that -implements +org.apache.accumulo.core.sample.Sampler+ along with options for -that class. For guidance on implementing a Sampler see that interface's -javadoc. Accumulo provides a few implementations out of the box. For -information on how to use the samplers that ship with Accumulo look in the -package `org.apache.accumulo.core.sample` and consult the javadoc of the -classes there. See the https://github.com/apache/accumulo-examples/blob/master/docs/sample.md[sampling example] -for examples of how to configure a Sampler on a table. - -Once a table is configured with a sampler all writes after that point will -generate sample data. For data written before sampling was configured sample -data will not be present. A compaction can be initiated that only compacts the -files in the table that do not have sample data. The example readme shows how -to do this. - -If the sampling configuration of a table is changed, then Accumulo will start -generating new sample data with the new configuration. However old data will -still have sample data generated with the previous configuration. A selective -compaction can also be issued in this case to regenerate the sample data. - -=== Scanning sample data - -Inorder to scan sample data, use the +setSamplerConfiguration(...)+ method on -+Scanner+ or +BatchScanner+. Please consult this methods javadocs for more -information. - -Sample data can also be scanned from within an Accumulo +SortedKeyValueIterator+. -To see how to do this, look at the example iterator referenced in the -https://github.com/apache/accumulo-examples/blob/master/docs/sample.md[sampling example]. -Also, consult the javadoc on +org.apache.accumulo.core.iterators.IteratorEnvironment.cloneWithSamplingEnabled()+. - -Map reduce jobs using the +AccumuloInputFormat+ can also read sample data. See -the javadoc for the +setSamplerConfiguration()+ method on -+AccumuloInputFormat+. - -Scans over sample data will throw a +SampleNotPresentException+ in the following cases : - -. sample data is not present, -. sample data is present but was generated with multiple configurations -. sample data is partially present - -So a scan over sample data can only succeed if all data written has sample data -generated with the same configuration. - -=== Bulk import - -When generating rfiles to bulk import into Accumulo, those rfiles can contain -sample data. To use this feature, look at the javadoc on the -+AccumuloFileOutputFormat.setSampler(...)+ method. - http://git-wip-us.apache.org/repos/asf/accumulo/blob/e99ec9f0/docs/src/main/asciidoc/chapters/security.txt ---------------------------------------------------------------------- diff --git a/docs/src/main/asciidoc/chapters/security.txt b/docs/src/main/asciidoc/chapters/security.txt deleted file mode 100644 index 4a57f8a..0000000 --- a/docs/src/main/asciidoc/chapters/security.txt +++ /dev/null @@ -1,182 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one or more -// contributor license agreements. See the NOTICE file distributed with -// this work for additional information regarding copyright ownership. -// The ASF licenses this file to You under the Apache License, Version 2.0 -// (the "License"); you may not use this file except in compliance with -// the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -== Security - -Accumulo extends the BigTable data model to implement a security mechanism -known as cell-level security. Every key-value pair has its own security label, stored -under the column visibility element of the key, which is used to determine whether -a given user meets the security requirements to read the value. This enables data of -various security levels to be stored within the same row, and users of varying -degrees of access to query the same table, while preserving data confidentiality. - -=== Security Label Expressions - -When mutations are applied, users can specify a security label for each value. This is -done as the Mutation is created by passing a ColumnVisibility object to the put() -method: - -[source,java] ----- -Text rowID = new Text("row1"); -Text colFam = new Text("myColFam"); -Text colQual = new Text("myColQual"); -ColumnVisibility colVis = new ColumnVisibility("public"); -long timestamp = System.currentTimeMillis(); - -Value value = new Value("myValue"); - -Mutation mutation = new Mutation(rowID); -mutation.put(colFam, colQual, colVis, timestamp, value); ----- - -=== Security Label Expression Syntax - -Security labels consist of a set of user-defined tokens that are required to read the -value the label is associated with. The set of tokens required can be specified using -syntax that supports logical AND +&+ and OR +|+ combinations of terms, as -well as nesting groups +()+ of terms together. - -Each term is comprised of one to many alpha-numeric characters, hyphens, underscores or -periods. Optionally, each term may be wrapped in quotation marks -which removes the restriction on valid characters. In quoted terms, quotation marks -and backslash characters can be used as characters in the term by escaping them -with a backslash. - -For example, suppose within our organization we want to label our data values with -security labels defined in terms of user roles. We might have tokens such as: - - admin - audit - system - -These can be specified alone or combined using logical operators: - ----- -// Users must have admin privileges -admin - -// Users must have admin and audit privileges -admin&audit - -// Users with either admin or audit privileges -admin|audit - -// Users must have audit and one or both of admin or system -(admin|system)&audit ----- - -When both +|+ and +&+ operators are used, parentheses must be used to specify -precedence of the operators. - -=== Authorization - -When clients attempt to read data from Accumulo, any security labels present are -examined against the set of authorizations passed by the client code when the -Scanner or BatchScanner are created. If the authorizations are determined to be -insufficient to satisfy the security label, the value is suppressed from the set of -results sent back to the client. - -Authorizations are specified as a comma-separated list of tokens the user possesses: - -[source,java] ----- -// user possesses both admin and system level access -Authorization auths = new Authorization("admin","system"); - -Scanner s = connector.createScanner("table", auths); ----- - -=== User Authorizations - -Each Accumulo user has a set of associated security labels. To manipulate -these in the shell while using the default authorizor, use the setuaths and getauths commands. -These may also be modified for the default authorizor using the java security operations API. - -When a user creates a scanner a set of Authorizations is passed. If the -authorizations passed to the scanner are not a subset of the users -authorizations, then an exception will be thrown. - -To prevent users from writing data they can not read, add the visibility -constraint to a table. Use the -evc option in the createtable shell command to -enable this constraint. For existing tables use the following shell command to -enable the visibility constraint. Ensure the constraint number does not -conflict with any existing constraints. - - config -t table -s table.constraint.1=org.apache.accumulo.core.security.VisibilityConstraint - -Any user with the alter table permission can add or remove this constraint. -This constraint is not applied to bulk imported data, if this a concern then -disable the bulk import permission. - -=== Pluggable Security - -New in 1.5 of Accumulo is a pluggable security mechanism. It can be broken into three actions -- -authentication, authorization, and permission handling. By default all of these are handled in -Zookeeper, which is how things were handled in Accumulo 1.4 and before. It is worth noting at this -point, that it is a new feature in 1.5 and may be adjusted in future releases without the standard -deprecation cycle. - -Authentication simply handles the ability for a user to verify their integrity. A combination of -principal and authentication token are used to verify a user is who they say they are. An -authentication token should be constructed, either directly through its constructor, but it is -advised to use the +init(Property)+ method to populate an authentication token. It is expected that a -user knows what the appropriate token to use for their system is. The default token is -+PasswordToken+. - -Once a user is authenticated by the Authenticator, the user has access to the other actions within -Accumulo. All actions in Accumulo are ACLed, and this ACL check is handled by the Permission -Handler. This is what manages all of the permissions, which are divided in system and per table -level. From there, if a user is doing an action which requires authorizations, the Authorizor is -queried to determine what authorizations the user has. - -This setup allows a variety of different mechanisms to be used for handling different aspects of -Accumulo's security. A system like Kerberos can be used for authentication, then a system like LDAP -could be used to determine if a user has a specific permission, and then it may default back to the -default ZookeeperAuthorizor to determine what Authorizations a user is ultimately allowed to use. -This is a pluggable system so custom components can be created depending on your need. - -=== Secure Authorizations Handling - -For applications serving many users, it is not expected that an Accumulo user -will be created for each application user. In this case an Accumulo user with -all authorizations needed by any of the applications users must be created. To -service queries, the application should create a scanner with the application -user's authorizations. These authorizations could be obtained from a trusted 3rd -party. - -Often production systems will integrate with Public-Key Infrastructure (PKI) and -designate client code within the query layer to negotiate with PKI servers in order -to authenticate users and retrieve their authorization tokens (credentials). This -requires users to specify only the information necessary to authenticate themselves -to the system. Once user identity is established, their credentials can be accessed by -the client code and passed to Accumulo outside of the reach of the user. - -=== Query Services Layer - -Since the primary method of interaction with Accumulo is through the Java API, -production environments often call for the implementation of a Query layer. This -can be done using web services in containers such as Apache Tomcat, but is not a -requirement. The Query Services Layer provides a mechanism for providing a -platform on which user facing applications can be built. This allows the application -designers to isolate potentially complex query logic, and enables a convenient point -at which to perform essential security functions. - -Several production environments choose to implement authentication at this layer, -where users identifiers are used to retrieve their access credentials which are then -cached within the query layer and presented to Accumulo through the -Authorizations mechanism. - -Typically, the query services layer sits between Accumulo and user workstations. http://git-wip-us.apache.org/repos/asf/accumulo/blob/e99ec9f0/docs/src/main/asciidoc/chapters/shell.txt ---------------------------------------------------------------------- diff --git a/docs/src/main/asciidoc/chapters/shell.txt b/docs/src/main/asciidoc/chapters/shell.txt deleted file mode 100644 index 4d53e3d..0000000 --- a/docs/src/main/asciidoc/chapters/shell.txt +++ /dev/null @@ -1,159 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one or more -// contributor license agreements. See the NOTICE file distributed with -// this work for additional information regarding copyright ownership. -// The ASF licenses this file to You under the Apache License, Version 2.0 -// (the "License"); you may not use this file except in compliance with -// the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -== Accumulo Shell -Accumulo provides a simple shell that can be used to examine the contents and -configuration settings of tables, insert/update/delete values, and change -configuration settings. - -The shell can be started by the following command: - - accumulo shell -u [username] - -The shell will prompt for the corresponding password to the username specified -and then display the following prompt: - - Shell - Apache Accumulo Interactive Shell - - - - version: 2.x.x - - instance name: myinstance - - instance id: 00000000-0000-0000-0000-000000000000 - - - - type 'help' for a list of available commands - - - root@myinstance> - -=== Basic Administration - -The Accumulo shell can be used to create and delete tables, as well as to configure -table and instance specific options. - ----- -root@myinstance> tables -accumulo.metadata -accumulo.root - -root@myinstance> createtable mytable - -root@myinstance mytable> - -root@myinstance mytable> tables -accumulo.metadata -accumulo.root -mytable - -root@myinstance mytable> createtable testtable - -root@myinstance testtable> - -root@myinstance testtable> deletetable testtable -deletetable { testtable } (yes|no)? yes -Table: [testtable] has been deleted. - -root@myinstance> ----- - -The Shell can also be used to insert updates and scan tables. This is useful for -inspecting tables. - ----- -root@myinstance mytable> scan - -root@myinstance mytable> insert row1 colf colq value1 -insert successful - -root@myinstance mytable> scan -row1 colf:colq [] value1 ----- - -The value in brackets ``[]'' would be the visibility labels. Since none were used, this is empty for this row. -You can use the +-st+ option to scan to see the timestamp for the cell, too. - -=== Table Maintenance - -The *compact* command instructs Accumulo to schedule a compaction of the table during which -files are consolidated and deleted entries are removed. - - root@myinstance mytable> compact -t mytable - 07 16:13:53,201 [shell.Shell] INFO : Compaction of table mytable started for given range - -The *flush* command instructs Accumulo to write all entries currently in memory for a given table -to disk. - - root@myinstance mytable> flush -t mytable - 07 16:14:19,351 [shell.Shell] INFO : Flush of table mytable - initiated... - -=== User Administration - -The Shell can be used to add, remove, and grant privileges to users. - ----- -root@myinstance mytable> createuser bob -Enter new password for 'bob': ********* -Please confirm new password for 'bob': ********* - -root@myinstance mytable> authenticate bob -Enter current password for 'bob': ********* -Valid - -root@myinstance mytable> grant System.CREATE_TABLE -s -u bob - -root@myinstance mytable> user bob -Enter current password for 'bob': ********* - -bob@myinstance mytable> userpermissions -System permissions: System.CREATE_TABLE -Table permissions (accumulo.metadata): Table.READ -Table permissions (mytable): NONE - -bob@myinstance mytable> createtable bobstable - -bob@myinstance bobstable> - -bob@myinstance bobstable> user root -Enter current password for 'root': ********* - -root@myinstance bobstable> revoke System.CREATE_TABLE -s -u bob ----- - -=== JSR-223 Support in the Shell - -The script command can be used to invoke programs written in languages supported by installed JSR-223 -engines. You can get a list of installed engines with the -l argument. Below is an example of the output -of the command when running the Shell with Java 7. - ----- -root@fake> script -l - Engine Alias: ECMAScript - Engine Alias: JavaScript - Engine Alias: ecmascript - Engine Alias: javascript - Engine Alias: js - Engine Alias: rhino - Language: ECMAScript (1.8) - Script Engine: Mozilla Rhino (1.7 release 3 PRERELEASE) -ScriptEngineFactory Info ----- - - A list of compatible languages can be found at https://en.wikipedia.org/wiki/List_of_JVM_languages. The -rhino javascript engine is provided with the JVM. Typically putting a jar on the classpath is all that is -needed to install a new engine. - - When writing scripts to run in the shell, you will have a variable called connection already available -to you. This variable is a reference to an Accumulo Connector object, the same connection that the Shell -is using to communicate with the Accumulo servers. At this point you can use any of the public API methods -within your script. Reference the script command help to see all of the execution options. Script and script -invocation examples can be found in ACCUMULO-1399.
