Thoudancer opened a new issue, #33494: URL: https://github.com/apache/doris/issues/33494
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version doris: 2.0.7 kafka: 2.3.0 ### What's Wrong? 1. I create a new topic with 5 partitions in kafka . At the begining, there is no data in the queue. 2. Then i create a table and a routine load to consume the topic data. 3. Show routine load, which shows the state is running, and the Progress is {"0":"OFFSET_END","1":"OFFSET_END","2":"OFFSET_END","3":"OFFSET_END","4":"OFFSET_END"}. 4. Push 3 msg to kafka topic, and show routine load again. The Process does't change, but the the lag of some partitions is increasing. Also clearly there isn't new data in the table. (if the default_offset is offset-end, it's normal.) 5. below is my routine load config ` sql = "CREATE ROUTINE LOAD {} ON {} " \ "COLUMNS(`{}`, event_date=from_unixtime({}/1000 , '%Y%m%d')) " \ "PROPERTIES ( " \ "\"desired_concurrent_number\" = \"2\", " \ "\"max_batch_interval\" = \"1\", " \ "\"max_batch_rows\" = \"200000\", " \ "\"max_batch_size\" = \"209715200\", " \ "\"max_error_number\" = \"3\", " \ "\"format\" = \"json\", " \ "\"strict_mode\" = \"false\", " \ "\"jsonpaths\" = \"{}\") " \ "FROM KAFKA ( " \ "\"kafka_broker_list\" = \"{}\", " \ "\"kafka_topic\" = \"{}\", " \ "\"property.client.id\" = \"{}\", " \ "\"property.group.id\" = \"{}\") "\` 6. what's more, when i increase the partition of an existing topic, the routine can find the new one, but the Process is also 'OFFSET_END', and the lag is increasing when i push new msg to new partition. ### What You Expected? the routine load can consume the data normally ### How to Reproduce? 1. I create a new topic with 5 partitions in kafka . At the begining, there is no data in the queue. 2. Then i create a table and a routine load to consume the topic data. 3. Show routine load, which shows the state is running, and the Progress is {"0":"OFFSET_END","1":"OFFSET_END","2":"OFFSET_END","3":"OFFSET_END","4":"OFFSET_END"}. ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
