[
https://issues.apache.org/jira/browse/KAFKA-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Bono updated KAFKA-16708:
-------------------------------
Description:
First of all, thank you all for adding the official Kafka Docker image (I know
it is a big responsibility and adds to the team workload).
I am migrating from {{wurstmeister/kafka}} to the official {{apache/kafka}}
image.
My advertised port is not static and was relying on [the PORT_COMMAND
feature|https://github.com/wurstmeister/kafka-docker/commit/c66375fc3b94e98dbecd603c5d2b44c06e927e88]
in the {{wurstmeister/kafka}} image to determine the port programatically.
This would let me define a docker-compose as follows:
{code:java}
services:
kafka:
image: apache/kafka:latest
hostname: kafka
ports:
- "9092"
volumes:
- '/var/run/docker.sock.raw:/var/run/docker.sock'
environment:
KAFKA_NODE_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
'CONTROLLER:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_LISTENERS:
'CONTROLLER://kafka:29093,PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_ADVERTISED_LISTENERS:
'PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://localhost:_{PORT_COMMAND}'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT_DOCKER'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
PORT_COMMAND: "docker ps | egrep 'kafka' | cut -d: -f 3 | cut -d- -f
1"{code}
Notice how the "ports" are dynamically mapped (i.e. not *"port:port"* syntax) -
the actual port will *not* be "9092".
Do you have a suggestion for an alternative approach on how to obtain a
non-static port for advertised listeners? If not, would adding conditional
support for this feature be welcomed?
I am aware of the complication/concern of this request as it is Docker in
Docker for non-root users (described
[here|https://jonfriesen.ca/articles/docker-in-docker-non-root-user/]) and as
such we could make it inactive by default and users would have to opt-in
explicitly.
I have created a [rough
WIP|https://github.com/onobc/kafka/commit/6556c4adbf08155b89c9804c2c5d1a988f8371f2]
that illustrates the concept (there is no conditionality in it currently).
Note that the container is not run as {*}root{*}, but rather the *appuser* is
added to whatever group that own the docker.sock (which on my machine is root).
P.S.
* This is my first time filing an issue w/ Kafka so if I missed anything
please let me know and I am glad to add whatever other info, etc..
* I am not sure what "Component" this should be under (the other Kafka Docker
related issues had differing values here)
was:
First of all, thank you all for adding the official Kafka Docker image (I know
it is a big responsibility and adds to the team workload).
I am migrating from {{wurstmeister/kafka}} to the official {{apache/kafka}}
image.
My advertised port is not static and was relying on [the PORT_COMMAND
feature|https://github.com/wurstmeister/kafka-docker/commit/c66375fc3b94e98dbecd603c5d2b44c06e927e88]
in the {{wurstmeister/kafka}} image to determine the port programatically.
This would let me define a docker-compose as follows:
{{}}
{code:java}
services:
kafka:
image: apache/kafka:latest
hostname: kafka
ports:
- "9092"
volumes:
- '/var/run/docker.sock.raw:/var/run/docker.sock'
environment:
KAFKA_NODE_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
'CONTROLLER:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_LISTENERS:
'CONTROLLER://kafka:29093,PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_ADVERTISED_LISTENERS:
'PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://localhost:_{PORT_COMMAND}'
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT_DOCKER'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
PORT_COMMAND: "docker ps | egrep 'kafka' | cut -d: -f 3 | cut -d- -f
1"{code}
{{}}
Notice how the "ports" are dynamically mapped (i.e. not *"port:port"* syntax) -
the actual port will *not* be "9092".
Do you have a suggestion for an alternative approach on how to obtain a
non-static port for advertised listeners? If not, would adding conditional
support for this feature be welcomed?
I am aware of the complication/concern of this request as it is Docker in
Docker for non-root users (described
[here|https://jonfriesen.ca/articles/docker-in-docker-non-root-user/]) and as
such we could make it inactive by default and users would have to opt-in
explicitly.
I have created a [rough
WIP|https://github.com/onobc/kafka/commit/6556c4adbf08155b89c9804c2c5d1a988f8371f2]
that illustrates the concept (there is no conditionality in it currently).
Note that the container is not run as {*}root{*}, but rather the *appuser* is
added to whatever group that own the docker.sock (which on my machine is root).
P.S. This is my first time filing an issue w/ Kafka so if I missed anything
please let me know and I am glad to add whatever other info, etc..
> Allow dynamic port for advertised listeners in Docker image
> -----------------------------------------------------------
>
> Key: KAFKA-16708
> URL: https://issues.apache.org/jira/browse/KAFKA-16708
> Project: Kafka
> Issue Type: Improvement
> Reporter: Chris Bono
> Priority: Major
>
> First of all, thank you all for adding the official Kafka Docker image (I
> know it is a big responsibility and adds to the team workload).
> I am migrating from {{wurstmeister/kafka}} to the official {{apache/kafka}}
> image.
> My advertised port is not static and was relying on [the PORT_COMMAND
> feature|https://github.com/wurstmeister/kafka-docker/commit/c66375fc3b94e98dbecd603c5d2b44c06e927e88]
> in the {{wurstmeister/kafka}} image to determine the port programatically.
> This would let me define a docker-compose as follows:
> {code:java}
> services:
> kafka:
> image: apache/kafka:latest
> hostname: kafka
> ports:
> - "9092"
> volumes:
> - '/var/run/docker.sock.raw:/var/run/docker.sock'
> environment:
> KAFKA_NODE_ID: 1
> KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
> 'CONTROLLER:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
> KAFKA_LISTENERS:
> 'CONTROLLER://kafka:29093,PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092'
> KAFKA_ADVERTISED_LISTENERS:
> 'PLAINTEXT_DOCKER://kafka:29092,PLAINTEXT_HOST://localhost:_{PORT_COMMAND}'
> KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
> KAFKA_PROCESS_ROLES: 'broker,controller'
> KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
> KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT_DOCKER'
> KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
> PORT_COMMAND: "docker ps | egrep 'kafka' | cut -d: -f 3 | cut -d- -f
> 1"{code}
> Notice how the "ports" are dynamically mapped (i.e. not *"port:port"* syntax)
> - the actual port will *not* be "9092".
> Do you have a suggestion for an alternative approach on how to obtain a
> non-static port for advertised listeners? If not, would adding conditional
> support for this feature be welcomed?
> I am aware of the complication/concern of this request as it is Docker in
> Docker for non-root users (described
> [here|https://jonfriesen.ca/articles/docker-in-docker-non-root-user/]) and as
> such we could make it inactive by default and users would have to opt-in
> explicitly.
> I have created a [rough
> WIP|https://github.com/onobc/kafka/commit/6556c4adbf08155b89c9804c2c5d1a988f8371f2]
> that illustrates the concept (there is no conditionality in it currently).
> Note that the container is not run as {*}root{*}, but rather the *appuser* is
> added to whatever group that own the docker.sock (which on my machine is
> root).
>
> P.S.
> * This is my first time filing an issue w/ Kafka so if I missed anything
> please let me know and I am glad to add whatever other info, etc..
> * I am not sure what "Component" this should be under (the other Kafka
> Docker related issues had differing values here)
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
