dujl opened a new pull request, #9752: URL: https://github.com/apache/incubator-doris/pull/9752
# Proposed changes Issue Number: https://github.com/apache/incubator-doris/issues/9557 Support query hudi external table in Doris. This pr support query cow and mor hudi table. When hudi table is a mor table, only support read optimized query mode. This is the second pr to support hudi external table. ## Problem Summary: Describe the overview of changes. The propose of the pr is: support query cow and mor hudi table. ### Design 1. generate scan range in Fe. Create a HudiScanNode to generate scan range parameters. HudiScanNode use HoodieParquetInputFormat to get all scan splits and assemble brokerRangeDesc 2. Scan hudi data in Be. we use broker_scan_node to scan parquet files that send by fe. 3. test case query mor table ```sql mysql> select * from t_hudi_mor; +---------------------+-----------------------+--------------------+------------------------+-----------------------------------------------------------------------------+------------+---------+-----------------------+----------------+ | _hoodie_commit_time | _hoodie_commit_seqno | _hoodie_record_key | _hoodie_partition_path | _hoodie_file_name | id | name | price | ts | +---------------------+-----------------------+--------------------+------------------------+-----------------------------------------------------------------------------+------------+---------+-----------------------+----------------+ | 20220522100249363 | 20220522100249363_0_1 | id:2 | | e9f6a051-2cfd-4cdd-94de-9d2d917dffb8-0_0-29-2007_20220522100249363.parquet | -729382776 | b1 | 4.63779796443726e-310 | 93870075838632 | | 20220522100310109 | 20220522100310109_0_2 | id:3 | | e9f6a051-2cfd-4cdd-94de-9d2d917dffb8-0_0-77-4028_20220522100310109.parquet | -729382456 | b1 | 4.63779796445307e-310 | 93870075838952 | | 20220522100324753 | 20220522100324753_0_3 | id:4 | | e9f6a051-2cfd-4cdd-94de-9d2d917dffb8-0_0-125-6049_20220522100324753.parquet | -729382136 | b1 | 4.63779796446888e-310 | 93870075839272 | | 20220522100339091 | 20220522100339091_0_4 | id:5 | | e9f6a051-2cfd-4cdd-94de-9d2d917dffb8-0_0-173-8070_20220522100339091.parquet | -729381816 | b1 | 4.63779796448469e-310 | 93870075839592 | | 20220522101838691 | 20220522101838691_0_1 | id:7 | | 018a955f-f1aa-4404-b997-e3abe72623fb-0_0-29-2009_20220522101838691.parquet | -729382776 | insert1 | 4.63779796443726e-310 | 93870075838632 | | 20220522101838691 | 20220522101838691_0_2 | id:6 | | 018a955f-f1aa-4404-b997-e3abe72623fb-0_0-29-2009_20220522101838691.parquet | -729382456 | insert2 | 4.63779796445307e-310 | 93870075838952 | | 20220520205326437 | 20220520205326437_0_2 | id:1 | | bbee0b74-9a04-45ae-b95d-12b448d82813-0_0-98-2079_20220520205326437.parquet | -729382776 | a1 | 4.63779796443726e-310 | 93870075838632 | +---------------------+-----------------------+--------------------+------------------------+-----------------------------------------------------------------------------+------------+---------+-----------------------+----------------+ ``` query cow partition table ```sql mysql> select * from t_hudi_cow_partition; +---------------------+-----------------------+--------------------+------------------------+-----------------------------------------------------------------------------+------------+--------+------------+------------+ | _hoodie_commit_time | _hoodie_commit_seqno | _hoodie_record_key | _hoodie_partition_path | _hoodie_file_name | id | name | price | ts | +---------------------+-----------------------+--------------------+------------------------+-----------------------------------------------------------------------------+------------+--------+------------+------------+ | | 20220520205249267_0_1 | id:1 | dt=2021-12-01 | b337c9d8-2088-4bfe-a30b-7923daa80958-0_0-3-0_20220520205249267.parquet | 1827790992 | a1 | 1827791008 | 1878892656 | | | 20220522192232625_0_1 | id:2 | dt=2022-5-20 | c70f8bd0-7871-4fe6-95c4-6fb80ca83a5d-0_0-30-2008_20220522192232625.parquet | -813514600 | name2 | -813514584 | -764204240 | | | 20220522192350827_0_2 | id:3 | dt=2022-5-21 | 27e3edf3-900f-45e9-adbc-b09d6b065aaf-0_0-82-4036_20220522192350827.parquet | 1878843544 | name3 | 1878843560 | 1878892752 | | | 20220522192429630_0_3 | id:31 | dt=2022-5-21 | 27e3edf3-900f-45e9-adbc-b09d6b065aaf-0_0-137-6071_20220522192429630.parquet | 1878843856 | name31 | 1878843872 | 1878892752 | | | 20220522192557945_0_4 | id:41 | dt=2022-5-22 | 60c2d68c-3318-4896-9af1-54f63324bd48-0_0-193-8109_20220522192557945.parquet | -815062888 | name41 | -815062872 | -764204144 | | | 20220522192557945_1_5 | id:51 | dt=2022-5-23 | 913fe002-b5ba-4106-8894-34a25fe2dbb3-0_1-199-8110_20220522192557945.parquet | 1827791000 | name51 | 1827791016 | 1878892848 | +---------------------+-----------------------+--------------------+------------------------+-----------------------------------------------------------------------------+------------+--------+------------+------------+ 6 rows in set (0.83 sec) ``` ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know)No 5. Has unit tests been added: (Yes/No/No Need)No Need 6. Has document been added or modified: (Yes/No/No Need)No Need 7. Does it need to update dependencies: (Yes/No)Yes 8. Are there any changes that cannot be rolled back: (Yes/No)No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
