kwonder0926 opened a new issue, #570: URL: https://github.com/apache/doris-flink-connector/issues/570
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version - Doris Version: 2.1.8 - Flink Version: 1.17.2 - Connector Version: flink-doris-connector-1.17-25.0.0.jar - MongoDB Version: 4.4.29 ### What's Wrong? Here is the command I used for synchronization. I'm performing a full-table synchronization: ```bash ./bin/flink run \ -Dexecution.checkpointing.interval=10s \ -Dparallelism.default=2 \ -Dpipeline.operator-chaining=false \ -c org.apache.doris.flink.tools.cdc.CdcTools \ ./lib/flink-doris-connector-1.17-25.0.0.jar \ mongodb-sync-database \ --job-name mongo2Doris \ --database xlsms_test \ --table-prefix db_ \ --schema-change-mode debezium_structure \ --mongodb-conf hosts=192.168.10.102:27017,192.168.10.103:27017,192.168.10.104:27017 \ --mongodb-conf username=stuser \ --mongodb-conf password=stpw \ --mongodb-conf database=xlsms \ --mongodb-conf scan.startup.mode=initial \ --mongodb-conf schema.sample-percent=0.2 \ --including-tables ".*" \ --sink-conf fenodes=192.168.10.102:8030 \ --sink-conf username=root \ --sink-conf password=123456 \ --sink-conf jdbc-url=jdbc:mysql://192.168.10.102:9030 \ --sink-conf sink.label-prefix=sms \ --sink-conf sink.enable-2pc=false \ --sink-conf sink.properties.format=json \ --sink-conf sink.properties.read_json_by_line=true \ --single-sink true \ --table-conf replication_num=1 ``` The synchronization process was normal. It successfully synchronized all the existing tables and data from my MongoDB database to Doris. However, when I created a new collection in MongoDB and inserted data into it, no new table appeared in Doris. My command for creating the collection and inserting data is as follows: ```javascript db["test"].insertMany([ { "name": "John Doe", "age": 30, "city": "New York" }, { "name": "Jane Smith", "age": 25, "city": "Los Angeles" } ]); ``` ### What You Expected? I expect that when I create a new collection with data in MongoDB, Doris can also synchronously create a corresponding table and synchronize the data. However, if I create an empty collection in MongoDB, Doris doesn't need to take any action. I've gone through the source code of the Doris connector on GitHub. Although I'm not entirely familiar with all the code logic, I suspect that the problem might be in the part where the connector listens for metadata changes in MongoDB. It may not be handling the CREATE COLLECTION events correctly, or there might be an issue with the logic that maps the new collection's schema to create a table in Doris. I'm looking forward to getting help on how to solve this problem. Is there a way to modify the source code to achieve the functionality I need? Or is there a configuration option that I've missed? Any advice or guidance would be greatly appreciated. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
