Mostafa Mokhtar created HADOOP-14558:
----------------------------------------
Summary: RPC requests on a secure cluster are 10x slower due to
expensive encryption and decryption
Key: HADOOP-14558
URL: https://issues.apache.org/jira/browse/HADOOP-14558
Project: Hadoop Common
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.6.0
Reporter: Mostafa Mokhtar
Priority: Critical
While running a performance tests for Impala comparing secure and un-secure
clusters I noticed that metadata loading operations are 10x slower on a cluster
with Kerberos+SSL enabled.
hadoop.rpc.protection is set to privacy
Any recommendations on how this can be mitigated? 10x slowdown is a big hit for
metadata loading.
The majority of the slowdown is coming from the two threads below.
{code}
Stack Trace Sample Count Percentage(%)
org.apache.hadoop.ipc.Client$Connection.run() 5,212 46.586
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse() 5,203 46.505
java.io.DataInputStream.readInt() 5,039 45.039
java.io.BufferedInputStream.read() 5,038 45.03
java.io.BufferedInputStream.fill() 5,038 45.03
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[], int, int)
5,036 45.013
java.io.FilterInputStream.read(byte[], int, int) 5,036
45.013
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.read(byte[], int,
int) 5,036 45.013
org.apache.hadoop.security.SaslRpcClient$WrappedInputStream.readNextRpcPacket()
5,035 45.004
com.sun.security.sasl.gsskerb.GssKrb5Base.unwrap(byte[], int, int) 4,775
42.68
sun.security.jgss.GSSContextImpl.unwrap(byte[],
int, int, MessageProp) 4,775 42.68
sun.security.jgss.krb5.Krb5Context.unwrap(byte[], int, int, MessageProp)
4,768 42.617
sun.security.jgss.krb5.WrapToken.getData()
4,714 42.134
sun.security.jgss.krb5.WrapToken.getData(byte[], int) 4,714 42.134
sun.security.jgss.krb5.WrapToken.getDataFromBuffer(byte[], int) 4,714
42.134
sun.security.jgss.krb5.CipherHelper.decryptData(WrapToken, byte[], int, int,
byte[], int) 3,083 27.556
sun.security.jgss.krb5.CipherHelper.des3KdDecrypt(WrapToken, byte[], int, int,
byte[], int) 3,078 27.512
sun.security.krb5.internal.crypto.Des3.decryptRaw(byte[], int, byte[], byte[],
int, int) 3,076 27.494
sun.security.krb5.internal.crypto.dk.DkCrypto.decryptRaw(byte[], int, byte[],
byte[], int, int) 3,076 27.494
{code}
And
{code}
Stack Trace Sample Count Percentage(%)
java.lang.Thread.run() 3,379 30.202
java.util.concurrent.ThreadPoolExecutor$Worker.run() 3,379 30.202
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
3,379 30.202
java.util.concurrent.FutureTask.run() 3,367 30.095
java.util.concurrent.Executors$RunnableAdapter.call() 3,367
30.095
org.apache.hadoop.ipc.Client$Connection$3.run() 3,367 30.095
java.io.DataOutputStream.flush() 3,367 30.095
java.io.BufferedOutputStream.flush() 3,367 30.095
java.io.BufferedOutputStream.flushBuffer() 3,367
30.095
org.apache.hadoop.security.SaslRpcClient$WrappedOutputStream.write(byte[], int,
int) 3,367 30.095
com.sun.security.sasl.gsskerb.GssKrb5Base.wrap(byte[], int, int) 3,281 29.326
sun.security.jgss.GSSContextImpl.wrap(byte[],
int, int, MessageProp) 3,281 29.326
sun.security.jgss.krb5.Krb5Context.wrap(byte[], int, int, MessageProp)
3,280 29.317
sun.security.jgss.krb5.WrapToken.<init>(Krb5Context, MessageProp, byte[], int,
int) 1,872 16.732
sun.security.jgss.krb5.WrapToken.encode() 1,407 12.576
{code}
This is the Impala Catalog thread which initiates the NameNode request
{code}
Stack Trace Sample Count Percentage(%)
org.apache.impala.service.JniCatalog.resetMetadata(byte[]) 2,414 21.577
org.apache.impala.service.CatalogOpExecutor.execResetMetadata(TResetMetadataRequest)
2,378 21.255
org.apache.impala.catalog.CatalogServiceCatalog.reloadTable(Table)
2,378 21.255
org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient,
Table) 2,351 21.014
org.apache.impala.catalog.HdfsTable.load(boolean, IMetaStoreClient,
Table, boolean, boolean, Set) 2,351 21.014
org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(IMetaStoreClient,
Set, boolean) 2,350 21.005
org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(List) 2,326
20.79
org.apache.impala.catalog.HdfsTable.loadPartitionFileMetadata(StorageDescriptor,
HdfsPartition) 2,233 19.959
org.apache.impala.catalog.HdfsTable.refreshFileMetadata(HdfsPartition) 1,998
17.858
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(Path) 1,496
13.371
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
