[GitHub] [incubator-doris] vagetablechicken edited a comment on issue #3469: [Memory Engine] Write Pipeline: internal data processing and storage

[GitBox]

vagetablechicken edited a comment on issue #3469:
URL: 
https://github.com/apache/incubator-doris/issues/3469#issuecomment-631289944


   ### WAL file format - need to rename, PartialRowBatch struct will change
   WAL file will save multiple partial rows. One write TX's WAL may be split by 
size => generates multiple {txn_id}_{segment_id}.wal
   
   For the related data structs, such as ParitalRowBatch, refer to 
https://github.com/decster/choco/blob/b644430e4e540da8ce0a6d13f07223c818706425/src/choco/partial_row_batch.h#L11
   
   A WAL file has the same struct as PartialRowBatch, shown below. WAL file 
size could be determined by config::write_buffer_size.
   
   
![image](https://user-images.githubusercontent.com/24697960/82416937-715a8980-9aad-11ea-8fb4-90a3fabb2176.png)
   
   The schema part, we could reuse the definition in segment proto.
   
https://github.com/apache/incubator-doris/blob/d2d95bfa841245d734bcd416b4c429ebe98b8321/gensrc/proto/segment_v2.proto#L158
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org