nextdreamblue opened a new issue, #14726: URL: https://github.com/apache/doris/issues/14726
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? right now, when query from aggregate table, VCollectIterator build a heap to sort all rows from rowsets, and set same if key column value is equal between rows. when there are many segments in one rowset, VMergeIterator build a heap to sort all rows from segment, but not set same if key column equal between rows. if you get many rows with same key from several segments of one rowset, VCollectIterator can not set same for all rows, _agg_key_next_block will get several rows with same key and not merge for agg. for example: table `dwd_fact_psd_order_details81` is a aggregate table, with key(`document_line_no`, `goods_no`, `company_code`, `document_no`, `dt`) MySQL [test]> select `document_line_no`, `goods_no`, `company_code`, `document_no`, `dt` from dwd_fact_psd_order_details81 where document_line_no='000010' and goods_no='xxxxxxxxxxxxx' and company_code = '2111' and document_no='0010000001' and dt='9999-12-30'; +------------------+-------------+--------------+-------------+------------+ | document_line_no | goods_no | company_code | document_no | dt | +------------------+-------------+--------------+-------------+------------+ | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | | 000010 | xxxxxxxxxxxxx | 2111 | 0010000001 | 9999-12-30 | +------------------+-------------+--------------+-------------+------------+ 8 rows in set (0.02 sec) ### What You Expected? expect only one row ### How to Reproduce? 大概通过如下步骤复现： 0、关闭自动compaction 1、写入一个tablet大批量的数据，这样rowset下有多个segment（这个地方需要大量数据，我测试环境把segment的大小设置为了几M） 2、一个rowset下的多个segment里有多条相同key的数据 读数据的时候触发这个逻辑： 以某个key进行查询，假定只从一个rowset中读取数据，按现在的代码，一个tablet内的数据理论是通过建立一个heap，对所有rowset下的数据，基于IteratorRowRef排序和相同key的标记（is_same = true).这个目前只用于排序从多个rowset读取的行。 假如key对应数据只从一个rowset读取，会触发不基于heap排序的逻辑，Level1Iterator只有一个child，不会走heap，直接从底层拿，这个时候没有标记相同key的逻辑。 而rowset内部,是从底层多个segment读取数据，走的是VMergeIterator，也会建立一个heap，这个地方也会基于key排序（但是没有相同key标记)。 基于以上过程获取到数据后，然后在_agg_key_next_block里，每一行数据都判断if (!_next_row.is_same)，如果相同则走聚合逻辑，但是在上边的读取逻辑下，is_same=false，这个地方认为两行key不同，并不会对相同key的行进行聚合，然后就返回了多行想通key的数据。 非向量化是在_agg_key_next_row这个地方现场对两row进行key比较，而不是在heap里通过is_same来标记，所以非向量化没问题。 另外上边只是一个出错的读取逻辑，当数据从多个rowset下读取时，也会存在问题。就算建立了heap去排序多个rowset下的key，并进行标记（is_same = true)，如果某个rowset下仍有多个相同key的行，仍然不能对所有row标记（is_same = true)，只会在_agg_key_next_block中对部分行进行聚合，仍返回多行的错误结果 ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
