Henry2SS opened a new issue, #17941: URL: https://github.com/apache/doris/issues/17941
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description ### Background Background Now doris uses BDBJE to process metadata log. In the scenario of high concurrent writing, there are performance problems. The write latency of BDBJE is between 1 ms and tens of ms, and only supports single concurrent write. The theoretical maximum TPS is around 1000. Therefore, there are serious performance problems in high-concurrency writing scenarios. ### Pressure test We tested doris' BDBJE with only one FE node on creating databases. In the locked state, the write TPS is about 800+; in the case of removing the lock, the write TPS is about 1500+. ### Industry Mainstream ZK writes TPS 30000+. RaftKeeper writes TPS at 70000+. [RaftKeeper Benchmark](https://github.com/JDRaftKeeper/RaftKeeper/blob/master/benchmark%2FBenchmark.md) ### The scheme to use raft #### Current situation of Doris FE：  #### The general idea of Raft management Log: 1. Implement the RaftCore module with C++, realize Doris FE log storage, and replace the original BDBJE. 2. The Java layer implements RaftJournal to implement the Journal interface. #### RaftCore Design: 1. Interface module: C++ RaftCore, adding, deleting, checking, member changing, and initialization and closing interface. Just provide a JNI interface. The write and read JNI interfaces support concurrent calls. 2.log module, state machine module, leader election #### RaftJournal Design: 1. When FE initializes RaftJournal, it calls the JNI interface to initialize the underlying C++ implementation The RaftCore. 2. RaftJournal internally uses the JNI interface to store and read metadata. 3. When closing FE, call the JNI interface to close the RaftCore implemented by the underlying C++. 4. Members change the JNI interface. ### Solution use RaftKeeper instead of BDBJE ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
