I'll ask about providing the executable. On Fri, May 9, 2025 at 1:52 PM John Mellor-Crummey <[email protected]> wrote:
> Hi Daniel, > > Thanks for the callstack. > > The problem seems to be exactly the same one recently encountered by Doug > Pase for a Fortran program at Sandia. This is a problem inside the type > processing by the Dyninst software written by our collaborators. > > Can you share a binary with us to facilitate debugging? The Sandia binary > is export controlled and only accessible inside their firewall. Having a > non-export controlled binary for debugging would make our lives easier. > > Best, > > John > -- > John Mellor-Crummey Professor > Dept of Computer Science Rice University > email: [email protected] phone: 713-348-5179 > > > > On May 9, 2025, at 12:15 PM, Daniel Kokron - NOAA Affiliate < > [email protected]> wrote: > > The application is compiled with Intel ifort. HPCToolkit and its > dependencies are compiled with gcc-13.2.1. I attached the spec for > HPCToolkit. > > > (gdb) run --nocache > /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x > Starting program: > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/hpctoolkit-2024.01.1-a3im66mlumyu3hbzmeuor3kj3l553yau/bin/hpcstruct > --nocache > /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x > Missing separate debuginfos, use: zypper install > glibc-debuginfo-2.31-150300.63.1.x86_64 > Missing separate debuginfo for > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libstdc++.so.6 > Try: zypper install -C > "debuginfo(build-id)=c74eca671e2dd0f063706372d103f8acef88f1e3" > Missing separate debuginfo for > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgomp.so.1 > Try: zypper install -C > "debuginfo(build-id)=54684492738e640bcd600e830cee025dd8771a20" > Missing separate debuginfo for > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgcc_s.so.1 > Try: zypper install -C > "debuginfo(build-id)=12f775ec4aeb94b749897b1b65638f18b61d1b1f" > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > begin sequential analysis of CPU binary > /lfs/h1/hpc/support/daniel.kokron/Tickets/2025042910000034/sorc/ufs_model.fd/build_fv3_1/gfs_model.x > (size = 377978672, threads = 1) > hpcstruct: > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/boost-1.87.0-2cldxfpwec5rbbhxutja5lwcgzh6fbhc/include/boost/smart_ptr/shared_ptr.hpp:550: > typename boost::detail::sp_member_access<T>::type > boost::shared_ptr<T>::operator->() const [with T = > Dyninst::SymtabAPI::typeSubrange; typename > boost::detail::sp_member_access<T>::type = > Dyninst::SymtabAPI::typeSubrange*]: Assertion `px != 0' failed. > > Program received signal SIGABRT, Aborted. > 0x0000155553e2fd2b in raise () from /lib64/libc.so.6 > (gdb) where > #0 0x0000155553e2fd2b in raise () from /lib64/libc.so.6 > #1 0x0000155553e313e5 in abort () from /lib64/libc.so.6 > #2 0x0000155553e27c6a in __assert_fail_base () from /lib64/libc.so.6 > #3 0x0000155553e27cf2 in __assert_fail () from /lib64/libc.so.6 > #4 0x0000155554d65127 in boost::enable_if<boost::integral_constant<bool, > !((bool)boost::is_same<Dyninst::SymtabAPI::Type, > Dyninst::SymtabAPI::typeSubrange>::value)>, > boost::shared_ptr<Dyninst::SymtabAPI::Type> >::type > Dyninst::SymtabAPI::typeCollection::addOrUpdateType<Dyninst::SymtabAPI::typeSubrange>(boost::shared_ptr<Dyninst::SymtabAPI::typeSubrange>) > () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #5 0x0000155554d547e6 in Dyninst::SymtabAPI::DwarfWalker::parseSubrange() > () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #6 0x0000155554d5a0a8 in > Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #7 0x0000155554d5b235 in > Dyninst::SymtabAPI::DwarfWalker::findAnyType(Dwarf_Attribute, bool, > boost::shared_ptr<Dyninst::SymtabAPI::Type>&) () > from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #8 0x0000155554d5b732 in > Dyninst::SymtabAPI::DwarfWalker::findType(boost::shared_ptr<Dyninst::SymtabAPI::Type>&, > bool) () > from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #9 0x0000155554d5497b in Dyninst::SymtabAPI::DwarfWalker::parseArray() () > from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #10 0x0000155554d59fb8 in > Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #11 0x0000155554d5a53b in > Dyninst::SymtabAPI::DwarfWalker::parse_int(Dwarf_Die, bool, bool) () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #12 0x0000155554d5bab0 in > Dyninst::SymtabAPI::DwarfWalker::parseModule(Dwarf_Die, > Dyninst::SymtabAPI::Module*&) [clone .constprop.0] [clone .isra.0] () > from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #13 0x0000155554d5c15c in Dyninst::SymtabAPI::DwarfWalker::parse() [clone > ._omp_fn.0] () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #14 0x000015555403b306 in GOMP_parallel () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/gcc-runtime-13.2.1-eo4evuugdi6s23do65dqomvbknlo4ong/lib/libgomp.so.1 > #15 0x0000155554d5d2ed in Dyninst::SymtabAPI::DwarfWalker::parse() () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #16 0x0000155554d0a0c1 in Dyninst::SymtabAPI::Object::parseTypeInfo() () > from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #17 0x0000155554cd48a7 in Dyninst::SymtabAPI::Symtab::parseTypes() () from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #18 0x0000155553fef5d7 in __pthread_once_slow () from > /lib64/libpthread.so.0 > #19 0x0000155554ccc8b4 in Dyninst::SymtabAPI::Symtab::parseTypesNow() () > from > /lfs/h1/hpc/support/daniel.kokron/SPACK/spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/dyninst-13.0.0-74gjdp5yt432nk3wyv7dn7o45ovdw6hr/lib/libsymtabAPI.so.13.0 > #20 0x00000000004418c8 in Inline::openSymtab (elfFile=elfFile@entry=0x8d94b0) > at Struct-Inline.cpp:132 > #21 0x000000000043cb31 in BAnal::Struct::makeStructure (filename=..., > outFile=outFile@entry=0x830ab0, gapsFile=gapsFile@entry=0x0, > gaps_filenm=..., search_path=..., structOpts=...) at Struct.cpp:770 > #22 0x000000000042cb42 in doSingleBinary (args=..., > sb=sb@entry=0x7ffffffd8740) > at /usr/include/c++/13/bits/basic_string.tcc:238 > #23 0x0000000000412cfd in realmain (argc=<optimized out>, argv=<optimized > out>) at main.cpp:209 > #24 0x000000000041220a in main (argc=<optimized out>, argv=<optimized > out>) at main.cpp:137 > > On Fri, May 9, 2025 at 11:16 AM John Mellor-Crummey <[email protected]> > wrote: > >> Hi Daniel, >> >> You should be able to run hpcstruct under gdb and then run it directly on >> the offending binary as follows >> >> gdb `which hpcstruct` >> run --nocache /path/to/gfs_model >> >> Then, you can send us a call path. By any chance is this a Fortran code >> compiled with gfortran? We are presently looking into a complaint about >> that from Sandia. >> >> Best, >> >> John >> -- >> John Mellor-Crummey Professor >> Dept of Computer Science Rice University >> email: [email protected] phone: 713-348-5179 >> >> >> >> On May 9, 2025, at 8:59 AM, Daniel Kokron - NOAA Affiliate via >> HPCToolkit-forum <[email protected]> wrote: >> >> I am encountering the following error while running hpcstruct. I cannot >> find the core file in any of the usual places. I have also tried running >> hpcstruct under gdb without getting very far. >> >> Wondering what my debugging options are? >> >> begin concurrent analysis of CPU binary gfs_model. (size = 377978416, >> threads = 1) >> /bin/sh: line 32: 63480 Aborted (core dumped) >> /spack/opt/spack/linux-sles15-zen2/gcc-13.2.1/hpctoolkit-2024.01.1-a3im66mlumyu3hbzmeuor3kj3l553yau/bin/hpcstruct. >> --nocache >> -j 1 -o $struct_name -M $meas_dir >> /Baseline_6Hr_WithWW3Restarts_Trace.16774.rawdata/cpubins/model.x > >> $warn_name 2>&1 >> >> Dan >> _______________________________________________ >> HPCToolkit-forum mailing list >> [email protected] >> https://mailman.rice.edu/mailman/listinfo/hpctoolkit-forum >> >> >> >
_______________________________________________ Dyninst-api mailing list [email protected] https://lists.cs.wisc.edu/mailman/listinfo/dyninst-api
