amorynan opened a new pull request, #18295:
URL: https://github.com/apache/doris/pull/18295
# Proposed changes
## Problem summary
When I insert data with bigger kv pair in one row for map , here coming a
core :
`WRITE of size 16 at 0x625055819400 thread T556 (BaseCompactionT)
#0 0x564829a56df3 in _mm_store_si128(long long __vector(2)*, long long
__vector(2))
/mnt/disk1/wangqiannan/tool/ldb_toolchain/lib/gcc/x86_64-linux-gnu/11/include/emmintrin.h:734
#1 0x564829a56df3 in inline_memcpy
/mnt/disk1/wangqiannan/amory/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:202
#2 0x564829a56eb0 in memcpy
/mnt/disk1/wangqiannan/amory/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
#3 0x564831035cb7 in doris::vectorized::ColumnVector<unsigned
char>::insert_range_from(doris::vectorized::IColumn const&, unsigned long,
unsigned long)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/columns/column_vector.cpp:362
#4 0x564830fe099c in
doris::vectorized::ColumnNullable::insert_range_from(doris::vectorized::IColumn
const&, unsigned long, unsigned long)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/columns/column_nullable.cpp:247
#5 0x564831222231 in
doris::vectorized::ColumnMap::insert_range_from(doris::vectorized::IColumn
const&, unsigned long, unsigned long)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/columns/column_map.cpp:255
#6 0x564830fe0a1e in
doris::vectorized::ColumnNullable::insert_range_from(doris::vectorized::IColumn
const&, unsigned long, unsigned long)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/columns/column_nullable.cpp:248
#7 0x56483e62a9d7 in
doris::vectorized::VerticalMergeIteratorContext::copy_rows(doris::vectorized::Block*,
unsigned long)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/olap/vertical_merge_iterator.cpp:267
#8 0x56483e631406 in
doris::vectorized::VerticalMaskMergeIterator::next_batch(doris::vectorized::Block*)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/olap/vertical_merge_iterator.cpp:585
#9 0x56483e64f082 in
doris::vectorized::VerticalBlockReader::_direct_next_block(doris::vectorized::Block*,
bool*)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/olap/vertical_block_reader.cpp:207
#10 0x564829d194fc in
doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*,
bool*)
/mnt/disk1/wangqiannan/amory/doris/be/src/vec/olap/vertical_block_reader.h:45
#11 0x564829cfe9d3 in
doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::Tablet>,
doris::ReaderType, std::shared_ptr<doris::TabletSchema>, bool,
std::vector<unsigned int, std::allocator<unsigned int> > const&,
doris::vectorized::RowSourcesBuffer*,
std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > > const&,
doris::RowsetWriter*, long, doris::Merger::Statistics*)
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/merger.cpp:216
#12 0x564829d01cd2 in
doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::Tablet>,
doris::ReaderType, std::shared_ptr<doris::TabletSchema>,
std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > > const&,
doris::RowsetWriter*, long, doris::Merger::Statistics*)
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/merger.cpp:323
#13 0x564829c9ff25 in doris::Compaction::do_compaction_impl(long)
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/compaction.cpp:287
#14 0x564829c9bab5 in doris::Compaction::do_compaction(long)
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/compaction.cpp:74
#15 0x56482a6ff2e0 in doris::BaseCompaction::execute_compact_impl()
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/base_compaction.cpp:76
#16 0x564829c9b606 in doris::Compaction::execute_compact()
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/compaction.cpp:55
#17 0x56482a7bf969 in
doris::Tablet::execute_compaction(doris::CompactionType)
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/tablet.cpp:1652
#18 0x564829c22097 in operator()
/mnt/disk1/wangqiannan/amory/doris/be/src/olap/olap_server.cpp:653
#19 0x564829c3177b in __invoke_impl<void,
doris::StorageEngine::_submit_compaction_task(doris::TabletSharedPtr,
doris::CompactionType)::<lambda()>&>
/mnt/disk1/wangqiannan/tool/ldb_toolchain/include/c++/11/bits/invoke.h:61
#20 0x564829c2e6f6 in __invoke_r<void,
doris::StorageEngine::_submit_compaction_task(doris::TabletSharedPtr,
doris::CompactionType)::<lambda()>&>
/mnt/disk1/wangqiannan/tool/ldb_toolchain/include/c++/11/bits/invoke.h:111
#21 0x564829c2c5ef in _M_invoke
/mnt/disk1/wangqiannan/tool/ldb_toolchain/include/c++/11/bits/std_function.h:291`
Describe your changes.
I calculate the pre-disk offsets data with offsets data in memory using
relative offset
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
