[
https://issues.apache.org/jira/browse/HBASE-28931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Connell updated HBASE-28931:
------------------------------------
Description:
At my company we have an issue with our HBase servers not reloading TLS
certificate files after they change on disk. We run our HMasters inside
Kubernetes Pods, and define our certificate contents as Kubernetes Secrets.
Then, the Secrets are projected into the HMaster containers as files. When the
value of a Secret changes, the file changes automatically. However, Kubernetes
does some complicated indirection, and does not change the files directly. It
swaps a new directory in with new files in it.
HBase sets up a WatchService on the directory containing the TLS cert. For
example, at my company, the cert is at
{{{}/etc/hadoop/conf/ssl/cert/server-chain.pem{}}}. Then, events from that
WatchService are delivered to a [handler
method|https://github.com/apache/hbase/blob/836630422df2776287a860eff9d7104c3eca0582/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/tls/X509Util.java#L530]
which contains this check:
{code:java}
Path eventFilePath = dirPath.resolve((Path) event.context());
if (filePath.equals(eventFilePath)) {
shouldResetContext = true;
}{code}
Debug logs show why this conditional is never true:
{code:java}
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,660 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.crypto.tls.X509Util: Ignoring watch event and
keeping previous default SSL context. Event kind: ENTRY_CREATE with context:
..2024_10_21_17_48_13.2471317370
.... {code}
The watcher events have a variety of files attached to them, but none of them
are {{{}server-chain.pem{}}}, so HBase thinks they are not relevant.
I propose that we simply remove the condition inspecting the file name that was
changed, and always reload the SSL context if a watcher event fires.
was:
At my company we have an issue with our HBase servers not reloading TLS
certificate files after they change on disk. We run our HMasters inside
Kubernetes Pods, and define our certificate contents as Kubernetes Secrets.
Then, the Secrets are projected into the HMaster containers as files. When the
value of a Secret changes, the file changes automatically. However, Kubernetes
does some complicated indirection, and does not change the files directly. It
swaps a new directory in with new files in it.
HBase sets up a WatchService on the directory containing the TLS cert. For
example, at my company, the cert is at
{{{}/etc/hadoop/conf/ssl/cert/server-chain.pem{}}}. Then, events from that
WatchService are delivered to a [handler
method|https://github.com/apache/hbase/blob/836630422df2776287a860eff9d7104c3eca0582/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/tls/X509Util.java#L530]
which contains this check:
{code:java}
Path eventFilePath = dirPath.resolve((Path) event.context());
if (filePath.equals(eventFilePath)) {
shouldResetContext = true;
}{code}
Debug logs show why this conditional is never true:
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
2024-10-21T17:48:13,660 [FileChangeWatcher-server-chain.pem] DEBUG
org.apache.hadoop.hbase.io.crypto.tls.X509Util: Ignoring watch event and
keeping previous default SSL context. Event kind: ENTRY_CREATE with context:
..2024_10_21_17_48_13.2471317370
....
The watch events have a variety of files attached to them, but none of them are
{{{}server-chain.pem{}}}, so HBase thinks they are not relevant.
I propose that we simply remove the condition inspecting the file name that was
changed, and always reload the SSL context if a watcher event fires.
> RPC TLS certificate is not reloaded when in Kubernetes Secret directory
> -----------------------------------------------------------------------
>
> Key: HBASE-28931
> URL: https://issues.apache.org/jira/browse/HBASE-28931
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Charles Connell
> Assignee: Charles Connell
> Priority: Major
>
> At my company we have an issue with our HBase servers not reloading TLS
> certificate files after they change on disk. We run our HMasters inside
> Kubernetes Pods, and define our certificate contents as Kubernetes Secrets.
> Then, the Secrets are projected into the HMaster containers as files. When
> the value of a Secret changes, the file changes automatically. However,
> Kubernetes does some complicated indirection, and does not change the files
> directly. It swaps a new directory in with new files in it.
> HBase sets up a WatchService on the directory containing the TLS cert. For
> example, at my company, the cert is at
> {{{}/etc/hadoop/conf/ssl/cert/server-chain.pem{}}}. Then, events from that
> WatchService are delivered to a [handler
> method|https://github.com/apache/hbase/blob/836630422df2776287a860eff9d7104c3eca0582/hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/tls/X509Util.java#L530]
> which contains this check:
> {code:java}
> Path eventFilePath = dirPath.resolve((Path) event.context());
> if (filePath.equals(eventFilePath)) {
> shouldResetContext = true;
> }{code}
> Debug logs show why this conditional is never true:
> {code:java}
> 2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
> org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
> ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
> 2024-10-21T17:48:13,659 [FileChangeWatcher-server-chain.pem] DEBUG
> org.apache.hadoop.hbase.io.FileChangeWatcher: Got file changed event:
> ENTRY_CREATE with context: ..2024_10_21_17_48_13.2471317370
> 2024-10-21T17:48:13,660 [FileChangeWatcher-server-chain.pem] DEBUG
> org.apache.hadoop.hbase.io.crypto.tls.X509Util: Ignoring watch event and
> keeping previous default SSL context. Event kind: ENTRY_CREATE with context:
> ..2024_10_21_17_48_13.2471317370
> .... {code}
> The watcher events have a variety of files attached to them, but none of them
> are {{{}server-chain.pem{}}}, so HBase thinks they are not relevant.
> I propose that we simply remove the condition inspecting the file name that
> was changed, and always reload the SSL context if a watcher event fires.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
