[I] How to build and package into jar file for Kafka Connect? [iceberg]

[via GitHub]

jesumyip opened a new issue, #12895:
URL: https://github.com/apache/iceberg/issues/12895

   ### Query engine
   
   Bytewax
   
   ### Question
   
   As the topic says. I have a docker compose yaml file set up with 2 
containers - Kafka and Kafka Connect.
   
   How do I build the iceberg-kafka-connect jar files and load them into the 
Kafka Connect container? I'm completely new to Java. I tried `../gradlew -xtest 
clean build` from within the `iceberg/kafka-connect/kafka-connect` folder. I 
then see a bunch of folders created under `build` folder. 
   
   
![Image](https://github.com/user-attachments/assets/eea3d174-0a59-4cbd-b27a-9fb2b3d93461)
   
   What do I do next? The only thing that somewhat makes sense to me is what's 
in the `libs` folder but I'm still unsure of what to do next.
   
   
![Image](https://github.com/user-attachments/assets/d3a9753e-a6cc-420e-af26-ff3f18cf6304)
   
   Here's the docker compose yaml I'm using (just showing the `connect` 
container). I also need this to talk to S3 as the file IO storage, and AWS Glue 
so I guess I need to package the AWS SDK JAR files as well into my host at 
`./custom-plugins` ? Don't I also need a `manifest.json` file?
   
   
   ```
   services:
     connect:
       build:
         context: .
         dockerfile: Dockerfile
       deploy:
         resources:
           limits:
             cpus: '2'
             memory: 2048M
       depends_on:
         kafka:
           condition: service_healthy
       ports:
         - 8083:8083
       environment:
         BOOTSTRAP_SERVERS: kafka:9092
         REST_ADVERTISED_HOST_NAME: connect
         REST_PORT: 8083
         GROUP_ID: compose-connect-group
         CONFIG_STORAGE_TOPIC: docker-connect-configs
         OFFSET_STORAGE_TOPIC: docker-connect-offsets
         STATUS_STORAGE_TOPIC: docker-connect-status
         KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
         VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
         CONFIG_STORAGE_REPLICATION_FACTOR: 1
         OFFSET_STORAGE_REPLICATION_FACTOR: 1
         STATUS_STORAGE_REPLICATION_FACTOR: 1
         CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY: All
         CONNECT_PLUGIN_PATH: "/opt/kafka/plugins,/opt/kafka/custom-plugins"
       healthcheck:
         test:
           [
             "CMD-SHELL",
             "curl --silent --fail http://connect:8083/connectors || exit 1",
           ]
         interval: 10s
         timeout: 5s
         retries: 5
       volumes:
         - ./data:/data
         - ./custom-plugins:/opt/kafka/custom-plugins
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org