JNSimba opened a new pull request, #168:
URL: https://github.com/apache/doris-flink-connector/pull/168
# Proposed changes
At present, the writing of Flink Connector can only rely on the writing of
checkpoint.
Not very friendly to two scenarios:
1. Jobs that do not need to enable checkpoint
2. The ETL of Flink jobs is complicated, which makes the checkpoint very
slow, but it does not want to affect the writing performance.
For these two situations, the batch write function is added:
Introduce parameters:
**sink.enable.batch-mode**: Whether to enable batch write mode, default is
false
**sink.flush.queue-size**: Queue length for async stream load, default is
2
**sink.buffer-flush.max-rows**: The maximum number of flush items in each
batch, the default is 5w
**sink.buffer-flush.max-bytes**: The maximum number of bytes flushed in each
batch, the default is 10MB
**sink.buffer-flush.interval**: the flush interval mills, over this time,
asynchronous threads will flush data. The default value is 1s.
FlinkSQL
```sql
CREATE TABLE doris_test_sink (
name STRING,
age INT
)
WITH (
'connector' = 'doris',
'fenodes' = 'xxx:8737',
'table.identifier' = 'test.test_flink',
'username' = 'root',
'password' = '',
'sink.enable.batch-mode' = 'true', --开启批量写入
'sink.buffer-flush.max-rows' = '50000', --最大的行数
'sink.buffer-flush.max-bytes' = '10485760',
'sink.buffer-flush.interval' = '5s',
'sink.properties.format' = 'json',
'sink.label-prefix' = 'label',
'sink.properties.read_json_by_line' = 'true'
)");
```
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
