[GitHub] [incubator-doris] vagetablechicken edited a comment on issue #3469: [Memory Engine] Write Pipeline: internal data processing and storage

[GitBox]

vagetablechicken edited a comment on issue #3469:
URL: 
https://github.com/apache/incubator-doris/issues/3469#issuecomment-631289944


   ### WAL file format - need to rename e.g.WriteTxnData?
   WAL file will save multiple partial rows. One write TX's WAL may be split by 
size => generates multiple {txn_id}_{segment_id}.wal
   
   For the related data structs, such as ParitalRowBatch, refer to 
https://github.com/decster/choco/blob/b644430e4e540da8ce0a6d13f07223c818706425/src/choco/partial_row_batch.h#L11
   
   A WAL file's `datas block` has the same struct as PartialRowBatch::_buffer, 
shown below. WAL file size could be determined by config::write_buffer_size.
   
   
![Lark20200603154131](https://user-images.githubusercontent.com/24697960/83609342-bce25c80-a5b0-11ea-9bd3-0ca1b45ef197.png)
   ```
   message FileFooterPB {
       optional uint32 version = 1 [default = 1]; // file version
       repeated ColumnMetaPB columns = 2; // tablet schema
       optional uint32 num_rows = 3; // number of values
   }
   ```
   
   The schema part(ColumnMetaPB), we could reuse the definition in segment 
proto.
   
https://github.com/apache/incubator-doris/blob/d2d95bfa841245d734bcd416b4c429ebe98b8321/gensrc/proto/segment_v2.proto#L158
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org