[
https://issues.apache.org/jira/browse/HBASE-28931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Connell updated HBASE-28931:
------------------------------------
Affects Version/s: 2.6.1
> RPC TLS certificate is not reloaded when in Kubernetes Secret directory
> -----------------------------------------------------------------------
>
> Key: HBASE-28931
> URL: https://issues.apache.org/jira/browse/HBASE-28931
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Charles Connell
> Assignee: Charles Connell
> Priority: Major
>
> At my company we have an issue with our HBase servers not reloading TLS
> certificate files after they change on disk. We run our HMasters inside
> Kubernetes Pods, and define our certificate contents as Kubernetes Secrets.
> Then, the Secrets are projected into the HMaster containers as files. When
> the value of a Secret changes, the file changes automatically. However,
> Kubernetes does some complicated indirection, and does not change the files
> directly. It swaps a new directory in with new files in it.
> HBase sets up a WatchService on the directory containing the TLS cert. For
> example, at my company, the cert is at
> {{{}/etc/hadoop/conf/ssl/cert/server-chain.pem{}}}. Then, events from that
> WatchService are delivered to a [handler
> method|https://github.com/apache/hbase/blob/836630422df2776287a860eff9d7104c3eca0582/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/tls/X509Util.java#L530]
> which contains this check:
> {code:java}
> Path eventFilePath = dirPath.resolve((Path) event.context());
> if (filePath.equals(eventFilePath)) {
> shouldResetContext = true;
> }{code}
> Debug logs show why this conditional is never true:
> 2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
> org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
> ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
> 2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
> org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
> ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
> 2024-10-21T17:48:13,660 [FileChangeWatcher-server-chain.pem] DEBUG
> org.apache.hadoop.hbase.io.crypto.tls.X509Util: Ignoring watch event and
> keeping previous default SSL context. Event kind: ENTRY_CREATE with context:
> ..2024_10_21_17_48_13.2471317370
> ....
> The watch events have a variety of files attached to them, but none of them
> are {{{}server-chain.pem{}}}, so HBase thinks they are not relevant.
> I propose that we simply remove the condition inspecting the file name that
> was changed, and always reload the SSL context if a watcher event fires.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
