Re:Re:Re: [Discuss] Prepare to release 1.1
Hi all, The outer join bug has been fixed in dev-1.0.1. And also fix some other bugs. This thread is closed and I will start vote for the v1.1 release. -- 此致！Best Regards 陈明雨 Mingyu Chen Email: morning...@apache.org 在 2022-06-27 09:52:53，"陈明雨" 写道： >Hi all, >The dev-1.0.1 is now closed. I'm just waiting for one last PR to resolve the >outer join bug. > > > > >-- > >此致！Best Regards >陈明雨 Mingyu Chen > >Email: >morning...@apache.org > > > > > >在 2022-06-24 10:06:05，"zhg yang" 写道： >>+1 we should maintain a version with bug fix but without new features >>Thanks >>Yang Zhengguo >> >> >>陈明雨 于2022年6月2日周四 23:27写道： >> >>> Hi all, >>> It has been more than two months since version 1.0 was released. >>> During this period, the Doris community has updated a lot of functions, >>> and made a lot of bug fixes and stability and performance improvements. >>> And version 1.0 also has some serious bug that need to be fixed. >>> So I recommend starting work on the 1.1 release. And a final release date >>> would be June 15th. >>> >>> >>> Version 1.1 will be produced from the dev-1.0.1[1] branch, mainly as an >>> optimized version of version 1.0, which fixes and optimizes a large number >>> of vectorization engine-related problems. >>> Compared with version 1.0, both performance and stability are greatly >>> improved. >>> >>> >>> All changes related to version 1.1 can be seen here [2]. At the same time, >>> we also have some PRs waiting to be merged in this version[3]. >>> And I drafted a Release Note[4]. >>> >>> >>> If you have a PR that needs to be merged into this release, please reply >>> in this thread, or simply label the PR with dev-1.0.1. >>> >>> >>> [1] https://github.com/apache/incubator-doris/tree/dev-1.0.1 >>> [2] >>> https://github.com/apache/incubator-doris/issues?q=is%3Apr+is%3Aclosed+label%3Adev%2Fmerged-1.0.1 >>> [3] >>> https://github.com/apache/incubator-doris/pulls?q=is%3Apr+is%3Aopen+label%3Adev%2F1.0.1 >>> [4] https://github.com/apache/incubator-doris/issues/9949 >>> >>> >>> >>> >>> -- >>> >>> 此致！Best Regards >>> 陈明雨 Mingyu Chen >>> >>> Email: >>> chenmin...@apache.org
Re:Re:Re:Re: [Discuss] Prepare to release 1.1
The first release candidate tag is: https://github.com/apache/doris/tree/1.1.0-rc01 -- 此致！Best Regards 陈明雨 Mingyu Chen Email: morning...@apache.org 在 2022-06-29 15:05:44，"Mingyu Chen" 写道： >Hi all, >The outer join bug has been fixed in dev-1.0.1. >And also fix some other bugs. > > >This thread is closed and I will start vote for the v1.1 release. > > > > >-- > >此致！Best Regards >陈明雨 Mingyu Chen > >Email: >morning...@apache.org > > > > >在 2022-06-27 09:52:53，"陈明雨" 写道： >>Hi all, >>The dev-1.0.1 is now closed. I'm just waiting for one last PR to resolve the >>outer join bug. >> >> >> >> >>-- >> >>此致！Best Regards >>陈明雨 Mingyu Chen >> >>Email: >>morning...@apache.org >> >> >> >> >> >>在 2022-06-24 10:06:05，"zhg yang" 写道： >>>+1 we should maintain a version with bug fix but without new features >>>Thanks >>>Yang Zhengguo >>> >>> >>>陈明雨 于2022年6月2日周四 23:27写道： >>> Hi all, It has been more than two months since version 1.0 was released. During this period, the Doris community has updated a lot of functions, and made a lot of bug fixes and stability and performance improvements. And version 1.0 also has some serious bug that need to be fixed. So I recommend starting work on the 1.1 release. And a final release date would be June 15th. Version 1.1 will be produced from the dev-1.0.1[1] branch, mainly as an optimized version of version 1.0, which fixes and optimizes a large number of vectorization engine-related problems. Compared with version 1.0, both performance and stability are greatly improved. All changes related to version 1.1 can be seen here [2]. At the same time, we also have some PRs waiting to be merged in this version[3]. And I drafted a Release Note[4]. If you have a PR that needs to be merged into this release, please reply in this thread, or simply label the PR with dev-1.0.1. [1] https://github.com/apache/incubator-doris/tree/dev-1.0.1 [2] https://github.com/apache/incubator-doris/issues?q=is%3Apr+is%3Aclosed+label%3Adev%2Fmerged-1.0.1 [3] https://github.com/apache/incubator-doris/pulls?q=is%3Apr+is%3Aopen+label%3Adev%2F1.0.1 [4] https://github.com/apache/incubator-doris/issues/9949 -- 此致！Best Regards 陈明雨 Mingyu Chen Email: chenmin...@apache.org
????streamload??????????????abort reason: coordinate BE is down
?? FE??3??BE ??streamload ?? BE ?? ?? W0625 21:11:04.788328 19871 stream_load_executor.cpp:281] commit transaction failed, errmsg=errCode = 2, detailMessage = transaction [1638] is already aborted. abort reason: coordinate BE is down, id=d14f60f37fe29182-e2452ee346c991b8, job_id=-1, txn_id=1638, label=xxx_t_3e9877180e1f415686443124a9fb72b0, elapse(s)=34 W0625 21:11:04.788368 19871 stream_load.cpp:144] handle streaming load failed, id=d14f60f37fe29182-e2452ee346c991b8, errmsg=errCode = 2, detailMessage = transaction [1638] is already aborted. abort reason: coordinate BE is down W0625 21:11:04.796619 19890 stream_load_executor.cpp:281] commit transaction failed, errmsg=errCode = 2, detailMessage = transaction [1640] is already aborted. abort reason: coordinate BE is down, id=bb4ac386050f365f-5697ff9a7614929d, job_id=-1, txn_id=1640, label=xxx_t_30bde0223a4c49bfae8855ea510e5157, elapse(s)=34 W0625 21:11:04.796648 19890 stream_load.cpp:144] handle streaming load failed, id=bb4ac386050f365f-5697ff9a7614929d, errmsg=errCode = 2, detailMessage = transaction [1640] is already aborted. abort reason: coordinate BE is down W0625 21:11:04.796726 19872 stream_load_executor.cpp:281] commit transaction failed, errmsg=errCode = 2, detailMessage = transaction [1632] is already aborted. abort reason: coordinate BE is down, id=3e4bc3cd3e3e853f-cf8604ac349eb1b5, job_id=-1, txn_id=1632, label=xxx_t_a20283afde854b54b362b91141571445, elapse(s)=34 W0625 21:11:04.796748 19894 stream_load_executor.cpp:281] commit transaction failed, errmsg=errCode = 2, detailMessage = transaction [1635] is already aborted. abort reason: coordinate BE is down, id=0c4a7e043b34eb88-9d27887e66e1e392, job_id=-1, txn_id=1635, label=xxx_t_638ea9d5da994462b56674f4218aa95d, elapse(s)=34 W0625 21:11:04.796767 19872 stream_load.cpp:144] handle streaming load failed, id=3e4bc3cd3e3e853f-cf8604ac349eb1b5, errmsg=errCode = 2, detailMessage = transaction [1632] is already aborted. abort reason: coordinate BE is down W0625 21:11:04.796799 19894 stream_load.cpp:144] handle streaming load failed, id=0c4a7e043b34eb88-9d27887e66e1e392, errmsg=errCode = 2, detailMessage = transaction [1635] is already aborted. abort reason: coordinate BE is down W0626 21:07:20.208528 19894 stream_load_executor.cpp:281] commit transaction failed, errmsg=errCode = 2, detailMessage = transaction [1834] is already aborted. abort reason: coordinate BE is down, id=6c4fc4d91eead9d1-c7bd542607b427a5, job_id=-1, txn_id=1834, label=xxx_t_a9c563f5eb2f45ee8a299889704ee42e, elapse(s)=33 W0626 21:07:20.208611 19894 stream_load.cpp:144] handle streaming load failed, id=6c4fc4d91eead9d1-c7bd542607b427a5, errmsg=errCode = 2, detailMessage = transaction [1834] is already aborted. abort reason: coordinate BE is down
[Discuss][DSIP] Support buffering data and automatically committing in Be to improve Doris real-time push ability
Hi all, In current data import ways，users generally use StreamLoad to push real-time data to Doris, which need to buffer multiple records in the client side, and then send them to Doris together. It is pretty complex for users. We propose to buffer the data in the server side(BE) and commit automatically so that it is transparent to users and users can insert records row by row. I'll add detailed design later.
Spark????Doris 1.0???????????????? Unrecognized field "keysType"
??
??doris 1.0 1??FE,3??BE.
sparkdoris
org.apache.doris.spark.exception.DorisException: Doris FE's response cannot map
to schema. res:
{"keysType":"DUP_KEYS","properties":[{"name":"id","aggregation_type":"","comment":"
id","type":"BIGINT"},{"name":"name","aggregation_type":"NONE","comment":"
","type":"VARCHAR"}],"status":200}
at
org.apache.doris.spark.rest.RestService.parseSchema(RestService.java:303)
at org.apache.doris.spark.rest.RestService.getSchema(RestService.java:279)
at
org.apache.doris.spark.sql.SchemaUtils$.discoverSchemaFromFe(SchemaUtils.scala:51)
at
org.apache.doris.spark.sql.SchemaUtils$.discoverSchema(SchemaUtils.scala:41)
at
org.apache.doris.spark.sql.DorisRelation.lazySchema$lzycompute(DorisRelation.scala:48)
at
org.apache.doris.spark.sql.DorisRelation.lazySchema(DorisRelation.scala:48)
at org.apache.doris.spark.sql.DorisRelation.schema(DorisRelation.scala:52)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:402)
at
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at
com.vxdata.datacenter.dataquality.util.SparkSqlUtil$.loadTable(SparkSqlUtil.scala:178)
at com.vxdata.buildmodel.computer.input.DBInput.compute(DBInput.scala:11)
at
com.vxdata.buildmodel.computer.service.JobFlowService.invokeCompute(JobFlowService.scala:239)
at
com.vxdata.buildmodel.computer.service.JobFlowService$$anonfun$exec$1.apply(JobFlowService.scala:81)
at
com.vxdata.buildmodel.computer.service.JobFlowService$$anonfun$exec$1.apply(JobFlowService.scala:78)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
com.vxdata.buildmodel.computer.service.JobFlowService.exec(JobFlowService.scala:78)
at
com.vxdata.buildmodel.computer.service.JobFlowService.exec(JobFlowService.scala:34)
at
com.vxdata.etl.SparkETLMain$$anonfun$1.apply$mcV$sp(SparkETLMain.scala:64)
at
com.vxdata.datacenter.dataquality.common.TApplication$class.start(TApplication.scala:42)
at com.vxdata.etl.SparkETLMain$.start(SparkETLMain.scala:12)
at
com.vxdata.etl.SparkETLMain$.delayedEndpoint$com$vxdata$etl$SparkETLMain$1(SparkETLMain.scala:20)
at
com.vxdata.etl.SparkETLMain$delayedInit$body.apply(SparkETLMain.scala:12)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at com.vxdata.etl.SparkETLMain$.main(SparkETLMain.scala:12)
at com.vxdata.etl.SparkETLMain.main(SparkETLMain.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: org.codehaus.jackson.map.exc.UnrecognizedPropertyException:
Unrecognized field "keysType" (Class
org.apache.doris.spark.rest.models.Schema), not marked as ignorable
at [Source: java.io.StringReader@7a1fa00e; line: 1, column: 14] (through
reference chain: org.apache.doris.spark.rest.models.Schema["keysType"])
at
org.codehaus.jackson.map.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:53)
at
org.codehaus.jackson.map.deser.StdDeserializationContext.unknownFieldException(StdDeserializationContext.java:267)
at
org.codehaus.jackson.map.deser.std.StdDeserializer.reportUnknownProperty(StdDeserializer.java:673)
at
org.codehaus.jackson.map.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:659)
at
org.codehaus.jackson.map.deser.BeanDeserializer.handleUnknownProperty(BeanDeserializer.java:1365)
at
org.codehaus.jackson.map.deser.BeanDeserializer._handleUnknown(BeanDeserializer.java:725)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:703)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
at
org.apache.doris.spark.rest.RestService.parseSchema(RestService.java:295) ...
38 more ;
Re: 通过streamload导入数据报异常abort reason: coordinate BE is down
应该是你的be挂了 show backends 看看 james <23461...@qq.com.invalid>于2022年6月29日 周三15:44写道： > 您好！ > > > 系统的部署情况说明：一台FE，3台BE 部署在一个局域网内。 > > > 在通过streamload导入大量数据时候，报上面的错误！ > 请问一般在什么情况下可能出现这样的问题，有什么解决方案没。 > BE 的 日志内容如下： > > > W0625 21:11:04.788328 19871 stream_load_executor.cpp:281] commit > transaction failed, errmsg=errCode = 2, detailMessage = transaction [1638] > is already aborted. abort reason: coordinate BE is down, > id=d14f60f37fe29182-e2452ee346c991b8, job_id=-1, txn_id=1638, > label=xxx_t_3e9877180e1f415686443124a9fb72b0, elapse(s)=34 > W0625 21:11:04.788368 19871 stream_load.cpp:144] handle streaming load > failed, id=d14f60f37fe29182-e2452ee346c991b8, errmsg=errCode = 2, > detailMessage = transaction [1638] is already aborted. abort reason: > coordinate BE is down > W0625 21:11:04.796619 19890 stream_load_executor.cpp:281] commit > transaction failed, errmsg=errCode = 2, detailMessage = transaction [1640] > is already aborted. abort reason: coordinate BE is down, > id=bb4ac386050f365f-5697ff9a7614929d, job_id=-1, txn_id=1640, > label=xxx_t_30bde0223a4c49bfae8855ea510e5157, elapse(s)=34 > W0625 21:11:04.796648 19890 stream_load.cpp:144] handle streaming load > failed, id=bb4ac386050f365f-5697ff9a7614929d, errmsg=errCode = 2, > detailMessage = transaction [1640] is already aborted. abort reason: > coordinate BE is down > W0625 21:11:04.796726 19872 stream_load_executor.cpp:281] commit > transaction failed, errmsg=errCode = 2, detailMessage = transaction [1632] > is already aborted. abort reason: coordinate BE is down, > id=3e4bc3cd3e3e853f-cf8604ac349eb1b5, job_id=-1, txn_id=1632, > label=xxx_t_a20283afde854b54b362b91141571445, elapse(s)=34 > W0625 21:11:04.796748 19894 stream_load_executor.cpp:281] commit > transaction failed, errmsg=errCode = 2, detailMessage = transaction [1635] > is already aborted. abort reason: coordinate BE is down, > id=0c4a7e043b34eb88-9d27887e66e1e392, job_id=-1, txn_id=1635, > label=xxx_t_638ea9d5da994462b56674f4218aa95d, elapse(s)=34 > W0625 21:11:04.796767 19872 stream_load.cpp:144] handle streaming load > failed, id=3e4bc3cd3e3e853f-cf8604ac349eb1b5, errmsg=errCode = 2, > detailMessage = transaction [1632] is already aborted. abort reason: > coordinate BE is down > W0625 21:11:04.796799 19894 stream_load.cpp:144] handle streaming load > failed, id=0c4a7e043b34eb88-9d27887e66e1e392, errmsg=errCode = 2, > detailMessage = transaction [1635] is already aborted. abort reason: > coordinate BE is down > W0626 21:07:20.208528 19894 stream_load_executor.cpp:281] commit > transaction failed, errmsg=errCode = 2, detailMessage = transaction [1834] > is already aborted. abort reason: coordinate BE is down, > id=6c4fc4d91eead9d1-c7bd542607b427a5, job_id=-1, txn_id=1834, > label=xxx_t_a9c563f5eb2f45ee8a299889704ee42e, elapse(s)=33 > W0626 21:07:20.208611 19894 stream_load.cpp:144] handle streaming load > failed, id=6c4fc4d91eead9d1-c7bd542607b427a5, errmsg=errCode = 2, > detailMessage = transaction [1834] is already aborted. abort reason: > coordinate BE is down -- 张家峰
?????? ????streamload??????????????abort reason: coordinate BE is down
??BEload??BE?? -- -- ??: "dev"
Re: Re: [Discuss][DSIP] Support Merge-On-Write implementation for UNIQUE KEY data model
Updated the scheduling https://cwiki.apache.org/confluence/display/DORIS/DSIP-018%3A+Support+Merge-On-Write+implementation+for+UNIQUE+KEY+data+model Best Chen Zhang 在 2022年6月27日 +0800 11:59，Chen Zhang ，写道： > Hi Devs, I've update the DISP last weekend, if you are interest on this > feature, welcome to review and comment, thanks > https://cwiki.apache.org/confluence/display/DORIS/DSIP-018%3A+Support+Merge-On-Write+implementation+for+UNIQUE+KEY+data+model > > Best > Chen Zhang > 在 2022年6月24日 +0800 10:13，zhg yang ，写道： > > @ Chen Zhang For the more important features, it is best to send a DISP > > first to let everyone discuss the design > > Thanks > > Yang Zhengguo > > > > > > Chen Zhang 于2022年6月23日周四 22:30写道： > > > > > @Minghong We'll use a multi-version delete bitmap, only save delta for > > > each version. > > > For example, we have a rowset with version [0-98], transaction 99 updated > > > some row in that rowset, and so does transaction 100 and 101, there would > > > be 3 delete bitmaps on that rowset, corresponding to rows updated by > > > version 99, 100 and 101. A query with version x will only see the bitmap > > > up > > > to version x. There's more details about space saving and cache > > > acceleration, let's discuss it in DSIP. > > > > > > @Xiaoli, our team have finished most develop works for the basic function > > > in our private repository, but there‘s still lots of works to do, welcome > > > to get involve. > > > > > > @Mingyu, could you help to create a DISP doc? I don't seem to have > > > permission. > > > > > > Best > > > Chen Zhang > > > On Jun 23, 2022, 21:41 +0800, Zhou Minghong , > > > wrote: > > > > Hi Chen Zhang > > > > one question about "and a delete bitmap (marks some rowid as deleted)”： > > > > how to handle transaction information by a bitmap? > > > > for example, transaction_100 delete a row, but this still visible to > > > transaction_99, but not visible to trasanction_101. How to handle this > > > case? > > > > > > > > > > > > Br/Minghong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2022-06-23 19:14:58, "Zhu,Xiaoli" wrote: > > > > > Hi Chen Zhang, > > > > > > > > > > I am very interested in this topic, and want to participate in the > > > development. > > > > > > > > > > 在 2022/6/23 下午2:44，“Chen Zhang” 写入: > > > > > > > > > > Hi devs, > > > > > > > > > > Unique-Key data model is widely used in scenarios like Flink-CDC, user > > > > > profile(用户画像), E-commerce orders, but the query performance for > > > > > current > > > > > Merge-On-Read implementation is not good, due to the following > > > > > reasons: > > > > > > > > > > 1. Doris can't determine whether one row in a segment file is latest > > > > > or > > > > > outdated, so it has to do some extra merge sort before getting the > > > > > latest data, and key comparison is quite CPU-costive. > > > > > 2. Aggregate function predicate push down is not supported by the > > > > > Unique-Key data model due to reason(1). > > > > > > > > > > I'd like to propose to support a Merge-On-Write implementation for the > > > > > Unique-Key data model, which leverages a new segment-file-level > > > > > primary > > > > > key index (used for point lookup on write) and a delete bitmap (marks > > > some > > > > > rowid as deleted), which can optimize read performance significantly. > > > > > > > > > > At the beginning, we wanted to add another Primary-Key data model with > > > > > Merge-On-Write implementation, but after a lot of discussion, we'd > > > prefer > > > > > to improve the Unique-Key data model rather than adding another one. > > > > > > > > > > I'll add detailed design and related research in the DSIP doc later. > > > > > > > > > > > > > > > - > > > > > To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org > > > > > For additional commands, e-mail: dev-h...@doris.apache.org > > > > > > > >
Re:Re:Re:Re:Re: [Discuss] Prepare to release 1.1
There is a bug that may cause query stuck, which has been fix in master branch[1]. So I cherry-pick part of it to dev-1.0.1 branch[2] and make a new tag 1.1.0-rc02[3] [1] https://github.com/apache/doris/pull/10214 [2] https://github.com/apache/doris/commit/589b44eddf6eafcd55057468de828b2c222ceedd [3] https://github.com/apache/doris/tree/1.1.0-rc02 -- 此致！Best Regards 陈明雨 Mingyu Chen Email: morning...@apache.org 在 2022-06-29 15:09:25，"Mingyu Chen" 写道： >The first release candidate tag is: >https://github.com/apache/doris/tree/1.1.0-rc01 > > > > >-- > >此致！Best Regards >陈明雨 Mingyu Chen > >Email: >morning...@apache.org > > > > > >在 2022-06-29 15:05:44，"Mingyu Chen" 写道： >>Hi all, >>The outer join bug has been fixed in dev-1.0.1. >>And also fix some other bugs. >> >> >>This thread is closed and I will start vote for the v1.1 release. >> >> >> >> >>-- >> >>此致！Best Regards >>陈明雨 Mingyu Chen >> >>Email: >>morning...@apache.org >> >> >> >> >>在 2022-06-27 09:52:53，"陈明雨" 写道： >>>Hi all, >>>The dev-1.0.1 is now closed. I'm just waiting for one last PR to resolve the >>>outer join bug. >>> >>> >>> >>> >>>-- >>> >>>此致！Best Regards >>>陈明雨 Mingyu Chen >>> >>>Email: >>>morning...@apache.org >>> >>> >>> >>> >>> >>>在 2022-06-24 10:06:05，"zhg yang" 写道： +1 we should maintain a version with bug fix but without new features Thanks Yang Zhengguo 陈明雨 于2022年6月2日周四 23:27写道： > Hi all, > It has been more than two months since version 1.0 was released. > During this period, the Doris community has updated a lot of functions, > and made a lot of bug fixes and stability and performance improvements. > And version 1.0 also has some serious bug that need to be fixed. > So I recommend starting work on the 1.1 release. And a final release date > would be June 15th. > > > Version 1.1 will be produced from the dev-1.0.1[1] branch, mainly as an > optimized version of version 1.0, which fixes and optimizes a large number > of vectorization engine-related problems. > Compared with version 1.0, both performance and stability are greatly > improved. > > > All changes related to version 1.1 can be seen here [2]. At the same time, > we also have some PRs waiting to be merged in this version[3]. > And I drafted a Release Note[4]. > > > If you have a PR that needs to be merged into this release, please reply > in this thread, or simply label the PR with dev-1.0.1. > > > [1] https://github.com/apache/incubator-doris/tree/dev-1.0.1 > [2] > https://github.com/apache/incubator-doris/issues?q=is%3Apr+is%3Aclosed+label%3Adev%2Fmerged-1.0.1 > [3] > https://github.com/apache/incubator-doris/pulls?q=is%3Apr+is%3Aopen+label%3Adev%2F1.0.1 > [4] https://github.com/apache/incubator-doris/issues/9949 > > > > > -- > > 此致！Best Regards > 陈明雨 Mingyu Chen > > Email: > chenmin...@apache.org
Re:[Discuss][DSIP] Support buffering data and automatically committing in Be to improve Doris real-time push ability
Hi Meiyi, Thanks for you proposal. And for "buffer the data in the server side(BE)", Doris already support the "begin; insert; insert; insert; ...; commit" method, which is also a kind of "buffer data in server side" feature for "insert into values" operations. Maybe you can explain more about how it works with stream load and how to use it. Please register an account here[1] and tell me your account name, I will create a DSIP for this proposal. [1] https://cwiki.apache.org/confluence/display/DORIS/Home -- 此致！Best Regards 陈明雨 Mingyu Chen Email: morning...@apache.org At 2022-06-29 15:54:34, "梅祎" wrote: >Hi all, > >In current data import ways，users generally use StreamLoad to push >real-time data to Doris, which need to buffer multiple records in the >client side, and then send them to Doris together. It is pretty complex for >users. > >We propose to buffer the data in the server side(BE) and commit >automatically so that it is transparent to users and users can insert >records row by row. > >I'll add detailed design later.
Re:Re:[Discuss][DSIP] Support buffering data and automatically committing in Be to improve Doris real-time push ability
Will there be too much performance drop if client insert records row by row？ due to many rpcs？ At 2022-06-29 21:31:02, "Mingyu Chen" wrote: >Hi Meiyi, >Thanks for you proposal. >And for "buffer the data in the server side(BE)", Doris already support the >"begin; insert; insert; insert; ...; commit" method, >which is also a kind of "buffer data in server side" feature for "insert into >values" operations. > > >Maybe you can explain more about how it works with stream load and how to use >it. >Please register an account here[1] and tell me your account name, I will >create a DSIP for this proposal. > > >[1] https://cwiki.apache.org/confluence/display/DORIS/Home > > > > >-- > >此致！Best Regards >陈明雨 Mingyu Chen > >Email: >morning...@apache.org > > > > > >At 2022-06-29 15:54:34, "梅祎" wrote: >>Hi all, >> >>In current data import ways，users generally use StreamLoad to push >>real-time data to Doris, which need to buffer multiple records in the >>client side, and then send them to Doris together. It is pretty complex for >>users. >> >>We propose to buffer the data in the server side(BE) and commit >>automatically so that it is transparent to users and users can insert >>records row by row. >> >>I'll add detailed design later.
Re: [Discuss][DSIP] Support buffering data and automatically committing in Be to improve Doris real-time push ability
Hi Meiyi, Thanks for you proposal. I would like to know why doris need this thing and what actually it is. About what, it that you want to add something like kafka in Doris? About why, can you explain more in what scenarios does the user need this? 梅祎 于2022年6月29日周三 15:54写道： > Hi all, > > In current data import ways，users generally use StreamLoad to push > real-time data to Doris, which need to buffer multiple records in the > client side, and then send them to Doris together. It is pretty complex for > users. > > We propose to buffer the data in the server side(BE) and commit > automatically so that it is transparent to users and users can insert > records row by row. > > I'll add detailed design later. > -- 王博 Wang Bo
Re: [Discuss][DSIP] Support buffering data and automatically committing in Be to improve Doris real-time push ability
Hi all, thanks for your reply. Let me update details in DISP firstly. @Mingyu, my account is meiyi, thanks. 王博 于2022年6月30日周四 09:18写道： > Hi Meiyi, > Thanks for you proposal. > I would like to know why doris need this thing and what actually it is. > About what, it that you want to add something like kafka in Doris? > About why, can you explain more in what scenarios does the user need this? > > 梅祎 于2022年6月29日周三 15:54写道： > > > Hi all, > > > > In current data import ways，users generally use StreamLoad to push > > real-time data to Doris, which need to buffer multiple records in the > > client side, and then send them to Doris together. It is pretty complex > for > > users. > > > > We propose to buffer the data in the server side(BE) and commit > > automatically so that it is transparent to users and users can insert > > records row by row. > > > > I'll add detailed design later. > > > > > -- > 王博 Wang Bo >
Re:Re: [Discuss][DSIP] Support buffering data and automatically committing in Be to improve Doris real-time push ability
Done: https://cwiki.apache.org/confluence/display/DORIS/DSIP-020%3A+Support+buffering+data+and+automatically+committing+for+stream+load -- 此致！Best Regards 陈明雨 Mingyu Chen Email: morning...@apache.org 在 2022-06-30 11:31:15，"梅祎" 写道： >Hi all, thanks for your reply. Let me update details in DISP firstly. >@Mingyu, my account is meiyi, thanks. > >王博 于2022年6月30日周四 09:18写道： > >> Hi Meiyi, >> Thanks for you proposal. >> I would like to know why doris need this thing and what actually it is. >> About what, it that you want to add something like kafka in Doris? >> About why, can you explain more in what scenarios does the user need this? >> >> 梅祎 于2022年6月29日周三 15:54写道： >> >> > Hi all, >> > >> > In current data import ways，users generally use StreamLoad to push >> > real-time data to Doris, which need to buffer multiple records in the >> > client side, and then send them to Doris together. It is pretty complex >> for >> > users. >> > >> > We propose to buffer the data in the server side(BE) and commit >> > automatically so that it is transparent to users and users can insert >> > records row by row. >> > >> > I'll add detailed design later. >> > >> >> >> -- >> 王博 Wang Bo >>
