But, you said you need to remove libcmalloc *not* libtcmalloc... I saw librbd/librados is built with libcmalloc not with libtcmalloc.. So, are you saying to remove libtcmalloc (not libcmalloc) to enable jemalloc ?
-----Original Message----- From: Stefan Priebe [mailto:[email protected]] Sent: Wednesday, August 19, 2015 1:31 PM To: Somnath Roy; Alexandre DERUMIER; Mark Nelson Cc: ceph-devel Subject: Re: Ceph Hackathon: More Memory Allocator Testing Am 19.08.2015 um 22:29 schrieb Somnath Roy: > Hmm...We need to fix that as part of configure/Makefile I guess (?).. > Since we have done this jemalloc integration originally, we can take that > ownership unless anybody sees a problem of enabling tcmalloc/jemalloc with > librbd/librados. > > << You have to remove libcmalloc out of your build environment to get > this done How do I do that ? I am using Ubuntu and can't afford to remove > libc* packages. I always use a chroot to build packages where only a minimal bootstrap + the build deps are installed. googleperftools where libtcmalloc comes from is not Ubuntu "core/minimal". Stefan > > Thanks & Regards > Somnath > > -----Original Message----- > From: Stefan Priebe [mailto:[email protected]] > Sent: Wednesday, August 19, 2015 1:18 PM > To: Somnath Roy; Alexandre DERUMIER; Mark Nelson > Cc: ceph-devel > Subject: Re: Ceph Hackathon: More Memory Allocator Testing > > > Am 19.08.2015 um 22:16 schrieb Somnath Roy: >> Alexandre, >> I am not able to build librados/librbd by using the following config option. >> >> ./configure –without-tcmalloc –with-jemalloc > > Same issue to me. You have to remove libcmalloc out of your build environment > to get this done. > > Stefan > > >> It seems it is building osd/mon/Mds/RGW with jemalloc enabled.. >> >> root@emsnode10:~/ceph-latest/src# ldd ./ceph-osd >> linux-vdso.so.1 => (0x00007ffd0eb43000) >> libjemalloc.so.1 => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 >> (0x00007f5f92d70000) >> ....... >> >> root@emsnode10:~/ceph-latest/src/.libs# ldd ./librados.so.2.0.0 >> linux-vdso.so.1 => (0x00007ffed46f2000) >> libboost_thread.so.1.55.0 => >> /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.55.0 (0x00007ff687887000) >> liblttng-ust.so.0 => /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0 >> (0x00007ff68763d000) >> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff687438000) >> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >> (0x00007ff68721a000) >> libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so >> (0x00007ff686ee0000) >> libsmime3.so => /usr/lib/x86_64-linux-gnu/libsmime3.so >> (0x00007ff686cb3000) >> libnspr4.so => /usr/lib/x86_64-linux-gnu/libnspr4.so >> (0x00007ff686a76000) >> libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 >> (0x00007ff686871000) >> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff686668000) >> libboost_system.so.1.55.0 => >> /usr/lib/x86_64-linux-gnu/libboost_system.so.1.55.0 (0x00007ff686464000) >> libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 >> (0x00007ff686160000) >> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff685e59000) >> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff685a94000) >> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 >> (0x00007ff68587e000) >> liblttng-ust-tracepoint.so.0 => >> /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0 (0x00007ff685663000) >> liburcu-bp.so.1 => /usr/lib/liburcu-bp.so.1 (0x00007ff68545c000) >> liburcu-cds.so.1 => /usr/lib/liburcu-cds.so.1 (0x00007ff685255000) >> /lib64/ld-linux-x86-64.so.2 (0x00007ff68a0f6000) >> libnssutil3.so => /usr/lib/x86_64-linux-gnu/libnssutil3.so >> (0x00007ff685029000) >> libplc4.so => /usr/lib/x86_64-linux-gnu/libplc4.so >> (0x00007ff684e24000) >> libplds4.so => /usr/lib/x86_64-linux-gnu/libplds4.so >> (0x00007ff684c20000) >> >> It is building with libcmalloc always... >> >> Did you change the ceph makefiles to build librbd/librados with jemalloc ? >> >> Thanks & Regards >> Somnath >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Alexandre >> DERUMIER >> Sent: Wednesday, August 19, 2015 7:01 AM >> To: Mark Nelson >> Cc: ceph-devel >> Subject: Re: Ceph Hackathon: More Memory Allocator Testing >> >> Thanks Marc, >> >> Results are matching exactly what I have seen with tcmalloc 2.1 vs 2.4 vs >> jemalloc. >> >> and indeed tcmalloc, even with bigger cache, seem decrease over time. >> >> >> What is funny, is that I see exactly same behaviour client librbd side, with >> qemu and multiple iothreads. >> >> >> Switching both server and client to jemalloc give me best performance on >> small read currently. >> >> >> >> >> >> >> ----- Mail original ----- >> De: "Mark Nelson" <[email protected]> >> À: "ceph-devel" <[email protected]> >> Envoyé: Mercredi 19 Août 2015 06:45:36 >> Objet: Ceph Hackathon: More Memory Allocator Testing >> >> Hi Everyone, >> >> One of the goals at the Ceph Hackathon last week was to examine how to >> improve Ceph Small IO performance. Jian Zhang presented findings showing a >> dramatic improvement in small random IO performance when Ceph is used with >> jemalloc. His results build upon Sandisk's original findings that the >> default thread cache values are a major bottleneck in TCMalloc 2.1. To >> further verify these results, we sat down at the Hackathon and configured >> the new performance test cluster that Intel generously donated to the Ceph >> community laboratory to run through a variety of tests with different memory >> allocator configurations. I've since written the results of those tests up >> in pdf form for folks who are interested. >> >> The results are located here: >> >> http://nhm.ceph.com/hackathon/Ceph_Hackathon_Memory_Allocator_Testing. >> pdf >> >> I want to be clear that many other folks have done the heavy lifting here. >> These results are simply a validation of the many tests that other folks >> have already done. Many thanks to Sandisk and others for figuring this out >> as it's a pretty big deal! >> >> Side note: Very little tuning other than swapping the memory allocator and a >> couple of quick and dirty ceph tunables were set during these tests. It's >> quite possible that higher IOPS will be achieved as we really start digging >> into the cluster and learning what the bottlenecks are. >> >> Thanks, >> Mark >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to [email protected] More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" >> in the body of a message to [email protected] More majordomo >> info at http://vger.kernel.org/majordomo-info.html >> >> ________________________________ >> >> PLEASE NOTE: The information contained in this electronic mail message is >> intended only for the use of the designated recipient(s) named above. If the >> reader of this message is not the intended recipient, you are hereby >> notified that you have received this message in error and that any review, >> dissemination, distribution, or copying of this message is strictly >> prohibited. If you have received this communication in error, please notify >> the sender by telephone or e-mail (as shown above) immediately and destroy >> any and all copies of this message in your possession (whether hard copies >> or electronically stored copies). >> >> N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w > j:+v w j m zZ+ ݢj" !tml= >> > N r y b X ǧv ^ ){.n + z ]z {ay ʇڙ ,j f h z w j:+v w j m zZ+ ݢj" !tml= >
