[
https://issues.apache.org/jira/browse/KNOX-2990?focusedWorklogId=895634&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-895634
]
ASF GitHub Bot logged work on KNOX-2990:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Dec/23 09:49
Start Date: 14/Dec/23 09:49
Worklog Time Spent: 10m
Work Description: smolnar82 opened a new pull request, #826:
URL: https://github.com/apache/knox/pull/826
## What changes were proposed in this pull request?
Implemented the changes I listed in
[KNOX-2990](https://issues.apache.org/jira/browse/KNOX-2990):
- deprecated the following TSS implementations:
- AliasBasedTokenStateService
- ZookeeperTokenStateService
- JournalBasedTokenStateService
- implemented a DerbyDB storage that stores tokens in
`$DATA_DIR/security/tokens` (this time it's not yet encrypted)
- file permissions are set on that folder to `700` (only the owner can
access it)
- changed the default implementation in `TokenStateServiceFactory` to the
new DerbyDatabaseTSS
- implemenedt a new KnoxCLI command that migrates existing tokens from
credential stores to any JDBC-based TSS backend (tested it with the new DerbyDB
TSS; see below)
- integrated this new KnoxCLI command in a way such that it runs when Knox
Gateway is started: if token management is enabled, and the configured TSS
implementation is a migration target (currently it's true for any JDBC-based
TSS implementation)
- added some new `gateway-site.xml` properties:
- `gateway.knox.token.migration.skip `: ensures that the previous
automated step can be controlled (E.g. in case of unforeseen errors it can be
turned off). Defaults
- `gateway.knox.token.migration.archive.tokens `: indicates if migrated
tokesn should be archived in another credential store called `__tokens`.
Defaults to `false`.
- `gateway.knox.token.migration.include.expired.tokens`: whether expired
tokens should be migrated or skipped. Defaults to `false`.
- `gateway.knox.token.migration.verbose`: if true, migrated/skipped tokens
are added in the `[gateway|knoxcli].log` and, optionally, on the STDOUT (when
running the KnoxCLI tool manually). Defaults to `true`, because it's very
useful to have the chancee to cross-reference token IDs in case of error
debugging.
- `gateway.knox.token.migration.progress.count`: the number of tokens
after the token migration tool displays progress in the logs and, optionally,
on the STDOUT.
- modified the token generation page to accept the new DerbyDB TSS.
## How was this patch tested?
Configured Knox to have the `AliasBasedTSS` as the token state backend and
to allow unlimited token creation:
```
<property>
<name>gateway.service.tokenstate.impl</name>
<value>org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService</value>
</property>
<property>
<name>gateway.knox.token.limit.per.user</name>
<value>-1</value>
</property>
```
Generated 456 tokens with random expiration times (456x4=1824 aliases) then
stopped the Knox GW (to avoid the reaper thread removing expired tokens).
```
$ bin/knoxcli.sh list-alias | grep "items"
1827
```
Executed the new KnoxCLI command to confirm it only migrates anything if the
configured backend allows token migration:
```
$ bin/knoxcli.sh migrate-tokens --progressCount 15 --archiveMigrated true
This tool is meant to migrate tokens into a JDBC TokenStateService backend.
However, the currently configured one
(org.apache.knox.gateway.services.token.impl.AliasBasedTokenStateService) does
not fulfill this requirement!
```
Before running the new KnoxLCI command again, I commented out the
`gateway.service.tokenstate.impl` param in `gateway-site.xml` => the new
default, DerbyDBTSS, was in place.
Executed the command again:
```
$ bin/knoxcli.sh migrate-tokens --progressCount 15 --archiveMigrated true >
~/migrationResultWithArchival.txt
$ bin/knoxcli.sh list-alias | grep items
3 items.
$ bin/knoxcli.sh list-alias --cluster __tokens | grep items
1824 items.
$ cat ~/migrationResultWithArchival.txt
Migrating tokens from __gateway credential store into the configured
TokenStateService backend...
Loading token aliases from the __gateway credential store. This could take a
while.
Token aliased loaded in 178741 milliseconds
Processed 15 tokens in 102 milliseconds
Processed 30 tokens in 174 milliseconds
...
Processed 450 tokens in 2191 milliseconds
Processed 456 tokens in 2202 milliseconds
Archiving token aliases in the __tokens credential store...
Archived token related aliases in the __tokens credential store in 141849
millsieconds
Removing token aliases from the __gateway credential store...
Removed token related aliases from the __gateway credential store in 38
milliseconds
```
---
Repeated the generate/migration step, but this time without token archival:
```
$ bin/knoxcli.sh migrate-tokens --progressCount 15 >
~/migrationResultWithoutArchival.txt
$ cat ~/migrationResultWithoutArchival.txt
Migrating tokens from __gateway credential store into the configured
TokenStateService backend...
Loading token aliases from the __gateway credential store. This could take a
while.
Token aliased loaded in 182497 milliseconds
Processed 15 tokens in 160 milliseconds
Processed 30 tokens in 271 milliseconds
...
Processed 456 tokens in 1677 milliseconds
Removing token aliases from the __gateway credential store...
Removed token related aliases from the __gateway credential store in 61
milliseconds
```
I also tested the token migration tool integration during the Knox Gateway
startup. I removed the previously created data/security/tokens folder, switched
to AliasBasedTSS and created another 456 tokens. Then switched back to the
default DerbyDBTSS and started the Knox GW:
```
2023-12-14 10:14:18,653 INFO knox.gateway
(GatewayServer.java:logSysProp(227)) - System Property: user.name=sandormolnar
2023-12-14 10:14:18,659 INFO knox.gateway
(GatewayServer.java:logSysProp(227)) - System Property:
user.dir=/Users/sandormolnar/test/knoxGateway
2023-12-14 10:14:18,659 INFO knox.gateway
(GatewayServer.java:logSysProp(227)) - System Property:
java.runtime.name=OpenJDK Runtime Environment
2023-12-14 10:14:18,659 INFO knox.gateway
(GatewayServer.java:logSysProp(227)) - System Property:
java.runtime.version=1.8.0_282-bre_2021_01_20_16_37-b00
2023-12-14 10:14:18,659 INFO knox.gateway
(GatewayServer.java:logSysProp(227)) - System Property:
java.home=/usr/local/Cellar/openjdk@8/1.8.0+282/libexec/openjdk.jdk/Contents/Home/jre
...
2023-12-14 10:14:22,001 INFO knox.gateway
(AbstractServiceFactory.java:logServiceUsage(103)) - Using
org.apache.knox.gateway.services.token.impl.DerbyDBTokenStateService
implementation for TokenStateService
...
2023-12-14 10:14:24,932 INFO knox.gateway
(AbstractGatewayServices.java:start(60)) - Starting service:
org.apache.knox.gateway.services.token.impl.DerbyDBTokenStateService
2023-12-14 10:14:24,937 INFO token.state
(TokenMigrationTool.java:log(114)) - Loading token aliases from the __gateway
credential store. This could take a while.
2023-12-14 10:17:35,276 INFO token.state
(TokenMigrationTool.java:log(114)) - Token aliases loaded in 190342 milliseconds
2023-12-14 10:17:35,514 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
cdb286ff-2037-41ea-8918-c338675f92eb into the configured TokenStateService
backend.
2023-12-14 10:17:35,515 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = bd11f6bf-6ee4-4322-98ab-6797f8ac7d00
2023-12-14 10:17:35,562 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
5b8a754f-78c7-428a-95d6-ff6cff586b87 into the configured TokenStateService
backend.
2023-12-14 10:17:35,601 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
d0281d9c-a652-4d79-824e-7b53f4476ea2 into the configured TokenStateService
backend.
2023-12-14 10:17:35,634 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
a4d506d8-28aa-41be-8a45-ee9e3f7d109c into the configured TokenStateService
backend.
2023-12-14 10:17:35,634 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 7d7a24b1-fcba-4828-9bc4-a22623a78357
2023-12-14 10:17:35,660 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
0c52b28b-b6d3-411a-a8af-d148a05aac0a into the configured TokenStateService
backend.
2023-12-14 10:17:35,684 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
d94263f1-5e5b-4fe3-9d78-74a26caee446 into the configured TokenStateService
backend.
2023-12-14 10:17:35,684 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 72e7bb73-d53b-4e89-9a8b-d5c4a17a93b3
2023-12-14 10:17:35,684 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 7f89e3c5-4567-40c7-b713-17e54695cb6c
2023-12-14 10:17:35,685 INFO token.state
(TokenMigrationTool.java:log(114)) - Processed 10 tokens in 342 milliseconds
2023-12-14 10:17:35,708 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
3dbe9224-ac82-43f1-b383-efd930dcfb09 into the configured TokenStateService
backend.
2023-12-14 10:17:35,731 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
760c1ff9-49f6-4ba0-9114-2c0dd3cdc9d2 into the configured TokenStateService
backend.
2023-12-14 10:17:35,753 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
d67cc4ab-4de0-4f2d-8664-3e9dd721b634 into the configured TokenStateService
backend.
2023-12-14 10:17:35,775 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
49a55ad7-44f6-48cc-bd4a-3db5a2108029 into the configured TokenStateService
backend.
2023-12-14 10:17:35,775 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 853527ab-6c52-4d8c-8dd6-f7cf5c4f3b27
2023-12-14 10:17:35,776 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 681206a9-3335-4129-90c3-50f996f194c1
2023-12-14 10:17:35,798 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
f424005f-2ae0-498c-949b-760c880e4662 into the configured TokenStateService
backend.
2023-12-14 10:17:35,799 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 8ab5fc0f-02d6-44d6-89ec-d41b4825c29d
2023-12-14 10:17:35,829 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
772e0455-4913-48ff-b66d-422e7d624f43 into the configured TokenStateService
backend.
2023-12-14 10:17:35,856 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
8b24070d-11f6-47e9-8832-faaee12faed0 into the configured TokenStateService
backend.
2023-12-14 10:17:35,857 INFO token.state
(TokenMigrationTool.java:log(114)) - Processed 20 tokens in 515 milliseconds
2023-12-14 10:17:35,857 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 1fc77b80-c96d-4f3e-9841-cd2a2703e2db
2023-12-14 10:17:35,857 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 4dd4955e-1380-4f12-9b5d-49052e4c794b
2023-12-14 10:17:35,882 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
dfa2ca51-2f9c-4ebf-95c1-760c1ed23d89 into the configured TokenStateService
backend.
2023-12-14 10:17:35,908 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
48a363c5-7abc-4914-ab11-84bd8caf7423 into the configured TokenStateService
backend.
2023-12-14 10:17:35,909 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 3a5888a1-d44a-4f72-b100-bbeeccc11f2b
2023-12-14 10:17:35,909 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = ca972a16-0458-4a07-b547-26d2e8361a0e
2023-12-14 10:17:35,909 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 6d132cf8-9307-4b53-9931-7ba950529850
2023-12-14 10:17:35,909 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = cdbba2df-8884-4caf-a693-a1ea5268f48f
2023-12-14 10:17:35,934 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
326e575a-6fea-4cc8-957c-e5829d76f517 into the configured TokenStateService
backend.
2023-12-14 10:17:35,954 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
31f7daee-e134-49e8-a633-4ec9e38bce57 into the configured TokenStateService
backend.
2023-12-14 10:17:35,954 INFO token.state
(TokenMigrationTool.java:log(114)) - Processed 30 tokens in 612 milliseconds
...
2023-12-14 10:17:40,306 INFO token.state
(TokenMigrationTool.java:log(114)) - Processed 440 tokens in 4964 milliseconds
2023-12-14 10:17:40,306 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 50439317-05a7-42ef-8dbc-0f4a648d4c46
2023-12-14 10:17:40,306 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 95332d90-60db-4d54-9b3f-202d2de3f7b4
2023-12-14 10:17:40,306 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = bca3afa2-f1f0-44ae-8766-c938a624e949
2023-12-14 10:17:40,317 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
7d3347c9-c059-42f6-b663-ab8964654010 into the configured TokenStateService
backend.
2023-12-14 10:17:40,328 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
6a879abc-03c8-42d6-b990-d2d61d45291c into the configured TokenStateService
backend.
2023-12-14 10:17:40,339 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
c7a298bd-b3df-418c-b235-3ae90dcfd1b4 into the configured TokenStateService
backend.
2023-12-14 10:17:40,358 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
05a4d692-0618-4008-a7b2-981bcb75a5fe into the configured TokenStateService
backend.
2023-12-14 10:17:40,374 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
84143f53-11a2-4d94-8f86-84971a2d1905 into the configured TokenStateService
backend.
2023-12-14 10:17:40,388 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
69787bc1-2361-4e90-a219-bf132bea6cce into the configured TokenStateService
backend.
2023-12-14 10:17:40,403 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
7c855052-9a69-4f52-8ee0-f8b952ecb656 into the configured TokenStateService
backend.
2023-12-14 10:17:40,403 INFO token.state
(TokenMigrationTool.java:log(114)) - Processed 450 tokens in 5061 milliseconds
2023-12-14 10:17:40,418 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
9b1ee5aa-9600-4fac-9f48-f152c01dd94b into the configured TokenStateService
backend.
2023-12-14 10:17:40,438 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
c03c4748-98c6-4e83-a399-f520e8d579b3 into the configured TokenStateService
backend.
2023-12-14 10:17:40,455 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
5bbb60ba-7608-4437-b4cc-72c81097bf8a into the configured TokenStateService
backend.
2023-12-14 10:17:40,455 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = 6da78f70-22d2-4f52-b9e7-f274d2028468
2023-12-14 10:17:40,455 INFO token.state
(TokenMigrationTool.java:log(114)) - Skipping the migration of expired token
with ID = e7b5e8a2-829a-40a2-8cd4-150943d69788
2023-12-14 10:17:40,474 INFO token.state
(TokenMigrationTool.java:log(114)) - Migrated token
1419101b-3714-4e32-9a3b-98ffa46a0e3e into the configured TokenStateService
backend.
2023-12-14 10:17:40,474 INFO token.state
(TokenMigrationTool.java:log(114)) - Processed 456 tokens in 5132 milliseconds
2023-12-14 10:17:40,474 INFO token.state
(TokenMigrationTool.java:log(114)) - Removing token aliases from the __gateway
credential store...
2023-12-14 10:17:40,536 INFO token.state
(TokenMigrationTool.java:log(114)) - Removed token related aliases from the
__gateway credential store in 62 milliseconds
```
Issue Time Tracking
-------------------
Worklog Id: (was: 895634)
Remaining Estimate: 0h
Time Spent: 10m
> TokenStateService implementation cleanup
> ----------------------------------------
>
> Key: KNOX-2990
> URL: https://issues.apache.org/jira/browse/KNOX-2990
> Project: Apache Knox
> Issue Type: Task
> Components: Server
> Affects Versions: 2.0.0, 1.6.0, 1.6.1
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Critical
> Fix For: 2.1.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This issue is driven by a [DISCUSS] thread initiated on Knox's DEV mailing
> list [here|https://lists.apache.org/thread/fs9nkl6l45o330ttvgvqxj3jnxt63bcs].
> As a result of that discussion, the following needs to be implemented:
> * deprecate the following TSS implementations:
> ** AliasBasedTokenStateService
> ** ZookeeperTokenStateService
> ** JournalBasedTokenStateService
> * document the deprecation of these TSS implementations in v2.1.0 and
> highlight that they will be removed in the upcoming release (v2.2.0?).
> * implement a DerbyDB storage that will store tokens in
> {{$DATA_DIR/security/tokens}} (encrypted or not, it'll be decided later)
> * make sure appropriate file permissions are set on that folder
> * have the {{homepage}} topology configured with JDBC TSS pointing to this
> DerbyDB storage
> * implement a new KnoxCLI command that migrates existing tokens from
> credential stores to the DerbyDB storage
> * automate this new KnoxCLI command in a way such that it runs when Knox
> Gateway is started, token management is enabled, and DerbyDB storage is
> configured
> * ensure that the previous automated step can be controlled (E.g. in case of
> unforeseen errors it can be turned off)
> * document possible data replication scenarios when, in the case of HA
> deployments, existing tokens from one Knox node should be made available in
> other Knox node(s) and there is no other centralized RDBMS in use
> (PostgreSQL, MySQL for instance)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
