amorynan opened a new pull request, #17795:
URL: https://github.com/apache/doris/pull/17795
# Proposed changes
map offsets fix
## Problem summary
When compaction case, memory map offsets coming to same olap convertor
which is from 0 to 0+size
but it should be continue in different pages when in one segment writer .
eg :
last block with map offset : [3, 6, 8, ... 100]
this block with map offset : [5, 10, 15 ..., 100]
the same convertor should record last offset to make later coming offset
followed last offset.
so after convertor :
the current offset should [105, 110, 115, ... 200], then column writer just
call append_data() to make the right offset data append pages
If we do not changes , when compaction or some read situation , here would
come a core
```
I0310 09:50:41.223521 449227 thrift_server.cpp:355] ThriftServer 'heartbeat'
started on port: 9050
*** Query id: 0-0 ***
*** Aborted at 1678413042 (unix time) try "date -d @1678413042" if you are
using GNU date ***
*** Current BE git commitID: e80ae03 ***
*** SIGSEGV address not mapped to object (@0x2450) received by PID 449227
(TID 449817 OR 0x7f6f13626700) from PID 9296; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int,
siginfo_t*, void*) at /mnt/datadisk0/doris/be/src/common/signal_handler.h:413
1# 0x00007F7043B24AF9 in
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F7043B1D1EC in
/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so
4# 0x00007F70493C53C0 in /lib/x86_64-linux-gnu/libpthread.so.0
5# je_large_prof_info_get at ../src/large.c:287
6# je_free_default at ../src/jemalloc.c:3014
7# bshuf_trans_bit_elem_AVX_avx2 in
/mnt/datadisk1/Doris-Deploy/20230310010008-doris-master-e80ae0367/be/lib/doris_be
8# bshuf_compress_lz4_block_avx2 in
/mnt/datadisk1/Doris-Deploy/20230310010008-doris-master-e80ae0367/be/lib/doris_be
9# bshuf_blocked_wrap_fun_avx2 in
/mnt/datadisk1/Doris-Deploy/20230310010008-doris-master-e80ae0367/be/lib/doris_be
10# bshuf_compress_lz4_avx2 in
/mnt/datadisk1/Doris-Deploy/20230310010008-doris-master-e80ae0367/be/lib/doris_be
11# doris::segment_v2::BitshufflePageBuilder<(doris::FieldType)8>::finish()
at /mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/bitshuffle_page.h:146
12# doris::segment_v2::ScalarColumnWriter::finish_current_page() at
/mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:667
13# doris::segment_v2::ScalarColumnWriter::append_data(unsigned char
const**, unsigned long) at
/mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:518
14# doris::segment_v2::MapColumnWriter::append_data(unsigned char const**,
unsigned long) at
/mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:1044
15# doris::segment_v2::MapColumnWriter::append_nullable(unsigned char
const*, unsigned char const**, unsigned long) at
/mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:1027
16# doris::segment_v2::ColumnWriter::append(unsigned char const*, void
const*, unsigned long) at
/mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:404
17# doris::segment_v2::SegmentWriter::append_block(doris::vectorized::Block
const*, unsigned long, unsigned long) at
/mnt/datadisk0/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:322
18# doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block
const*, std::vector<unsigned int, std::allocator<unsigned int> > const&, bool,
unsigned int) at
/mnt/datadisk0/doris/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:96
19#
doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::Tablet>,
doris::ReaderType, std::shared_ptr<doris::TabletSchema>, bool,
std::vector<unsigned int, std::allocator<unsigned int> > const&,
doris::vectorized::RowSourcesBuffer*,
std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > > const&,
doris::RowsetWriter*, long, doris::Merger::Statistics*) at
/mnt/datadisk0/doris/be/src/olap/merger.cpp:219
20# doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::Tablet>,
doris::ReaderType, std::shared_ptr<doris::TabletSchema>,
std::vector<std::shared_ptr<doris::RowsetReader>,
std::allocator<std::shared_ptr<doris::RowsetReader> > > const&,
doris::RowsetWriter*, long, doris::Merger::Statistics*) at
/var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:122
21# doris::Compaction::do_compaction_impl(long) at
/mnt/datadisk0/doris/be/src/olap/compaction.cpp:289
22# doris::Compaction::do_compaction(long) at
/mnt/datadisk0/doris/be/src/olap/compaction.cpp:75
23# doris::CumulativeCompaction::execute_compact_impl() at
/mnt/datadisk0/doris/be/src/olap/cumulative_compaction.cpp:78
24# doris::Compaction::execute_compact() at
/mnt/datadisk0/doris/be/src/olap/compaction.cpp:56
25# doris::Tablet::execute_compaction(doris::CompactionType) at
/mnt/datadisk0/doris/be/src/olap/tablet.cpp:1597
26# std::_Function_handler<void (),
doris::StorageEngine::_submit_compaction_task(std::shared_ptr<doris::Tablet>,
doris::CompactionType)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at
/var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
27# doris::ThreadPool::dispatch_thread() at
/mnt/datadisk0/doris/be/src/util/threadpool.cpp:537
28# doris::Thread::supervise_thread(void*) at
/mnt/datadisk0/doris/be/src/util/thread.cpp:454
29# start_thread in /lib/x86_64-linux-gnu/libpthread.so.0
30# __clone in /lib/x86_64-linux-gnu/libc.so.6
```
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
