gaodayue commented on issue #3859: URL: https://github.com/apache/incubator-doris/issues/3859#issuecomment-645299411
The problem could also occur when follower's metadata lags behind master by a substantial time. We encountered one case recently and I'd like to share the details. First of all, master FE finishes txn quickly and normally every 5 second. ``` 2020-06-17 14:14:08,315 INFO 32 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10210] and version hash to [3776798707510922300] 2020-06-17 14:14:13,256 INFO 32 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10211] and version hash to [6996331758766475117] 2020-06-17 14:14:18,300 INFO 32 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10212] and version hash to [1122327101224378913] ``` However, one of the followers encounters some transient issue and its metadata lags behind master during 14:14~14:16. This causes some queries picking obsolete version 10210 before 14:15:47, which then fails with -230 err code. ``` 2020-06-17 14:14:52,896 INFO 81 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10210] and version hash to [3776798707510922300] 2020-06-17 14:15:47,789 INFO 81 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10211] and version hash to [6996331758766475117] 2020-06-17 14:16:36,344 INFO 81 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10212] and version hash to [1122327101224378913] 2020-06-17 14:16:36,407 INFO 81 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] set partition 264562638's version to [10213] and version hash to [4855950519590045391] ``` The proposed `latest_read_version` can't help in this case. I think MVVC is the way to go at the end of the day. > In fact, @chaoyli has written a function that supports multi-path lookup of version graph, maybe we can implement a complete MVCC logic on this basis. +1. What's the progress of this work? I'd like to help if possible. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
