Hi, I previously sent the question below to the user list, but didn't hear back, and after more testing I think it maybe a dev question... potentially specific to MemoryMappedSegment behavior.
Also, I see similar ext4 vs xfs performance difference on both Azure and GCP VMs running CentOS 7.5. >From additional tests, seeing that when using commitlog_compression (e.g. >LZ4Compressor) the performance of xfs and ext4 is equivalent and good (i.e. >compressed xfs appears to perform as well as compressed and non-compressed >ext4). I see that the write code-path in non-compressed scenario is using >MappedByteBuffer<https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/commitlog/MemoryMappedSegment.java> > (i.e. memory mapped file) while compressed >code-path<https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/commitlog/CompressedSegment.java> > is using regular FileChannel and ByteBuffer. Usually MappedByteBuffer should >perform better, but seems to perform worse in a multi-threaded scenario with >xfs on CentOS 7.5 Here is an example of what it looks like with "rate threads=64" [cid:99c00a63-7d7e-49cb-97a9-b5ed19951942] Any ideas on what can be causing this difference in performance for XFS when using the default not compressed commitlog (i.e. MappedByteBuffer code path)? Thanks, Arsen ________________________________ From: arsenv...@outlook.com Sent: Sunday, June 16, 2019 9:35 PM To: u...@cassandra.apache.org Subject: Write perf better when commitlog is on ext4 vs xfs Hi, On CentOS 7.5 (3.10.0-957.21.2.el7.x86_64), I am seeing better write performance when commitlog is using "ext4" versus "xfs" filesystem. It is the same local/ephemeral disk within an Azure VM and the only change I am making is switching the filesystem on the disk from "ext4" to "xfs". Is this expected? Any ideas on how to troubleshoot and tune the filesystem (or Linux or Cassandra) so that xfs performs same or better than ext4? Details below. OpenJDK 1.8.0 (same with Oracle JDK 1.8.0) Cassandra version: 3.11.4 cassandra.yaml has this default config: commitlog_sync: periodic commitlog_sync_period_in_ms: 10000 commitlog_segment_size_in_mb: 32 I am testing write performance using the following cassandra-stress command: cqlsh --execute "DROP KEYSPACE IF EXISTS keyspace1;" 10.0.0.4 cassandra-stress write no-warmup n=2000000 cl=LOCAL_QUORUM \ -rate threads=32 throttle=0/s \ -pop seq="1..10000000" -col size=FIXED\(818\) \ -node "10.0.0.4" \ -schema keyspace=keyspace1 \ -log interval=1 -mode native cql3 connectionsPerHost=128 ext4 mount and ops/s look like this: /dev/sdb1 on /mnt/resource type ext4 (rw,relatime,seclabel,data=ordered) Connected to cluster: avcassandra, max pending requests per connection 128, max connections per host 128 Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0 Created keyspaces. Sleeping 1s for propagation. Sleeping 2s... Running WRITE with 32 threads for 2000000 iteration Failed to connect over JMX; not collecting these stats type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb total, 749, 749, 749, 749, 14.4, 8.4, 43.5, 79.7, 107.0, 131.9, 1.0, 0.00000, 0, 0, 0, 0, 0, 0 total, 4776, 4027, 4027, 4027, 7.9, 4.9, 24.6, 44.8, 65.4, 76.7, 2.0, 0.47357, 0, 0, 0, 0, 0, 0 total, 11129, 6353, 6353, 6353, 5.0, 2.5, 16.4, 26.6, 40.8, 46.5, 3.0, 0.34745, 0, 0, 0, 0, 0, 0 total, 17547, 6418, 6418, 6418, 4.9, 2.2, 16.4, 31.8, 74.1, 83.6, 4.0, 0.25930, 0, 0, 0, 0, 0, 0 total, 28396, 10849, 10849, 10849, 3.0, 1.5, 11.4, 21.2, 37.9, 49.0, 5.0, 0.25513, 0, 0, 0, 0, 0, 0 total, 40808, 12412, 12412, 12412, 2.5, 1.5, 9.0, 19.5, 33.8, 68.0, 6.0, 0.23341, 0, 0, 0, 0, 0, 0 total, 53080, 12272, 12272, 12272, 2.6, 1.3, 10.0, 19.0, 60.8, 73.5, 7.0, 0.20377, 0, 0, 0, 0, 0, 0 total, 67025, 13945, 13945, 13945, 2.3, 1.3, 9.1, 17.7, 30.5, 40.8, 8.0, 0.18276, 0, 0, 0, 0, 0, 0 total, 86886, 19861, 19861, 19861, 1.6, 1.2, 2.9, 9.3, 42.7, 56.5, 9.0, 0.18765, 0, 0, 0, 0, 0, 0 total, 106184, 19298, 19298, 19298, 1.6, 1.2, 3.1, 10.8, 57.8, 61.5, 10.0, 0.17614, 0, 0, 0, 0, 0, 0 total, 128877, 22693, 22693, 22693, 1.4, 1.2, 2.6, 4.1, 14.7, 30.2, 11.0, 0.16866, 0, 0, 0, 0, 0, 0 total, 148331, 19454, 19454, 19454, 1.6, 1.3, 3.0, 5.7, 45.3, 47.7, 12.0, 0.15466, 0, 0, 0, 0, 0, 0 total, 156974, 8643, 8643, 8643, 1.6, 1.2, 3.3, 10.2, 24.6, 41.2, 13.0, 0.14813, 0, 0, 0, 0, 0, 0 total, 170654, 13680, 13680, 13680, 3.6, 1.3, 3.1, 9.2, 890.8, 896.5, 14.0, 0.39489, 0, 0, 0, 0, 0, 0 total, 187715, 17061, 17061, 17061, 1.9, 1.2, 4.5, 13.0, 27.2, 44.2, 15.0, 0.37273, 0, 0, 0, 0, 0, 0 total, 205718, 18003, 18003, 18003, 1.7, 1.3, 3.7, 11.3, 32.8, 53.4, 16.0, 0.35161, 0, 0, 0, 0, 0, 0 total, 212945, 7227, 7227, 7227, 1.8, 1.2, 3.2, 11.2, 71.3, 88.5, 17.0, 0.34545, 0, 0, 0, 0, 0, 0 total, 220825, 7880, 7880, 7880, 6.4, 1.2, 3.1, 11.1, 1200.6, 1209.0, 18.0, 0.42966, 0, 0, 0, 0, 0, 0 total, 241273, 20448, 20448, 20448, 1.5, 1.2, 3.1, 7.3, 23.0, 60.4, 19.0, 0.40131, 0, 0, 0, 0, 0, 0 total, 261889, 20616, 20616, 20616, 1.5, 1.2, 2.9, 8.0, 25.0, 33.5, 20.0, 0.37691, 0, 0, 0, 0, 0, 0 total, 282464, 20575, 20575, 20575, 1.5, 1.2, 2.7, 5.2, 68.6, 72.7, 21.0, 0.35456, 0, 0, 0, 0, 0, 0 total, 305429, 22965, 22965, 22965, 1.4, 1.2, 2.5, 3.8, 17.2, 23.8, 22.0, 0.33370, 0, 0, 0, 0, 0, 0 total, 326504, 21075, 21075, 21075, 1.5, 1.2, 2.9, 5.6, 17.9, 31.5, 23.0, 0.31637, 0, 0, 0, 0, 0, 0 total, 342653, 16149, 16149, 16149, 1.9, 1.3, 3.8, 13.2, 107.0, 124.4, 24.0, 0.30293, 0, 0, 0, 0, 0, 0 total, 360972, 18319, 18319, 18319, 1.7, 1.2, 3.8, 11.5, 36.1, 63.7, 25.0, 0.28988, 0, 0, 0, 0, 0, 0 total, 379092, 18120, 18120, 18120, 1.7, 1.2, 2.9, 8.2, 133.5, 161.1, 26.0, 0.27680, 0, 0, 0, 0, 0, 0 total, 400379, 21287, 21287, 21287, 1.5, 1.2, 2.9, 6.4, 30.8, 43.2, 27.0, 0.26454, 0, 0, 0, 0, 0, 0 total, 422330, 21951, 21951, 21951, 1.4, 1.2, 2.6, 4.3, 21.9, 35.3, 28.0, 0.25311, 0, 0, 0, 0, 0, 0 total, 442491, 20161, 20161, 20161, 1.4, 1.2, 2.7, 5.0, 21.6, 41.5, 29.0, 0.24331, 0, 0, 0, 0, 0, 0 total, 456035, 13544, 13544, 13544, 11.9, 1.3, 3.4, 9.7, 4366.3, 4366.3, 34.0, 0.25639, 0, 0, 0, 0, 0, 0 total, 458788, 2753, 2753, 2753, 1.5, 1.3, 2.9, 4.2, 32.8, 40.9, 35.0, 0.25612, 0, 0, 0, 0, 0, 0 total, 468346, 9558, 9558, 9558, 46.4, 1.4, 7.1, 15.6, 13203.7, 13228.8, 48.0, 0.27131, 0, 0, 0, 0, 0, 0 total, 489463, 21117, 21117, 21117, 1.5, 1.2, 2.8, 8.5, 19.7, 32.8, 49.0, 0.26143, 0, 0, 0, 0, 0, 0 total, 510484, 21021, 21021, 21021, 1.5, 1.2, 2.9, 5.7, 20.2, 28.0, 50.0, 0.25232, 0, 0, 0, 0, 0, 0 total, 532895, 22411, 22411, 22411, 1.4, 1.2, 2.5, 4.8, 28.4, 40.7, 51.0, 0.24335, 0, 0, 0, 0, 0, 0 total, 553656, 20761, 20761, 20761, 1.5, 1.2, 3.0, 9.0, 18.2, 33.0, 52.0, 0.23551, 0, 0, 0, 0, 0, 0 xfs mount and ops/s look like this: /dev/sdb1 on /mnt/resource type xfs (rw,relatime,seclabel,attr2,inode64,noquota) Connected to cluster: avcassandra, max pending requests per connection 128, max connections per host 128 Datatacenter: dc1; Host: /10.0.0.4; Rack: rack0 Created keyspaces. Sleeping 1s for propagation. Sleeping 2s... Running WRITE with 32 threads for 2000000 iteration Failed to connect over JMX; not collecting these stats type total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb total, 1550, 1550, 1550, 1550, 14.9, 12.2, 37.8, 61.5, 75.6, 82.2, 1.0, 0.00000, 0, 0, 0, 0, 0, 0 total, 5742, 4192, 4192, 4192, 7.6, 4.6, 23.4, 36.7, 63.0, 78.8, 2.0, 0.32434, 0, 0, 0, 0, 0, 0 total, 10026, 4284, 4284, 4284, 7.4, 5.1, 20.4, 33.5, 95.2, 112.1, 3.0, 0.22200, 0, 0, 0, 0, 0, 0 total, 15802, 5776, 5776, 5776, 5.5, 3.2, 17.0, 32.8, 50.5, 62.6, 4.0, 0.18920, 0, 0, 0, 0, 0, 0 total, 23728, 7926, 7926, 7926, 4.0, 2.6, 11.6, 23.2, 54.7, 90.6, 5.0, 0.19722, 0, 0, 0, 0, 0, 0 total, 32143, 8415, 8415, 8415, 3.7, 2.5, 11.8, 20.4, 40.8, 48.1, 6.0, 0.17557, 0, 0, 0, 0, 0, 0 total, 41506, 9363, 9363, 9363, 3.4, 2.0, 11.0, 26.1, 72.2, 84.2, 7.0, 0.16341, 0, 0, 0, 0, 0, 0 total, 52696, 11190, 11190, 11190, 2.8, 2.6, 5.0, 10.8, 20.1, 32.3, 8.0, 0.15527, 0, 0, 0, 0, 0, 0 total, 63707, 11011, 11011, 11011, 2.9, 2.6, 4.6, 11.2, 20.7, 62.6, 9.0, 0.14386, 0, 0, 0, 0, 0, 0 total, 74246, 10539, 10539, 10539, 3.0, 2.7, 5.2, 11.9, 41.4, 46.3, 10.0, 0.13055, 0, 0, 0, 0, 0, 0 total, 84693, 10447, 10447, 10447, 3.0, 2.8, 5.0, 7.2, 13.6, 30.0, 11.0, 0.11879, 0, 0, 0, 0, 0, 0 total, 95304, 10611, 10611, 10611, 3.0, 2.8, 5.0, 7.2, 14.6, 32.6, 12.0, 0.10912, 0, 0, 0, 0, 0, 0 total, 105865, 10561, 10561, 10561, 3.0, 2.7, 4.8, 10.4, 53.1, 60.6, 13.0, 0.10111, 0, 0, 0, 0, 0, 0 total, 116431, 10566, 10566, 10566, 3.0, 2.8, 5.0, 8.9, 14.8, 25.2, 14.0, 0.09380, 0, 0, 0, 0, 0, 0 total, 127856, 11425, 11425, 11425, 2.8, 2.6, 4.5, 6.8, 16.6, 24.4, 15.0, 0.08816, 0, 0, 0, 0, 0, 0 total, 139136, 11280, 11280, 11280, 2.8, 2.6, 4.4, 6.5, 17.6, 20.6, 16.0, 0.08292, 0, 0, 0, 0, 0, 0 total, 149943, 10807, 10807, 10807, 2.9, 2.7, 4.5, 6.1, 50.0, 52.2, 17.0, 0.07817, 0, 0, 0, 0, 0, 0 total, 161338, 11395, 11395, 11395, 2.8, 2.6, 4.8, 10.1, 18.4, 30.0, 18.0, 0.07411, 0, 0, 0, 0, 0, 0 total, 172127, 10789, 10789, 10789, 2.9, 2.6, 5.3, 12.3, 41.9, 53.9, 19.0, 0.07025, 0, 0, 0, 0, 0, 0 total, 182078, 9951, 9951, 9951, 3.2, 2.1, 10.9, 20.9, 36.3, 80.7, 20.0, 0.06662, 0, 0, 0, 0, 0, 0 total, 193140, 11062, 11062, 11062, 2.9, 2.5, 5.3, 12.0, 29.0, 44.7, 21.0, 0.06354, 0, 0, 0, 0, 0, 0 total, 203536, 10396, 10396, 10396, 3.1, 2.6, 5.2, 11.7, 60.6, 68.7, 22.0, 0.06061, 0, 0, 0, 0, 0, 0 total, 214575, 11039, 11039, 11039, 2.9, 2.6, 5.0, 10.1, 22.5, 36.2, 23.0, 0.05801, 0, 0, 0, 0, 0, 0 total, 225072, 10497, 10497, 10497, 3.0, 2.6, 5.6, 13.2, 29.2, 48.2, 24.0, 0.05553, 0, 0, 0, 0, 0, 0 total, 235155, 10083, 10083, 10083, 3.1, 2.7, 4.8, 9.9, 89.3, 102.6, 25.0, 0.05329, 0, 0, 0, 0, 0, 0 total, 245257, 10102, 10102, 10102, 3.1, 2.6, 5.2, 11.4, 65.6, 67.3, 26.0, 0.05118, 0, 0, 0, 0, 0, 0 total, 256802, 11545, 11545, 11545, 2.7, 2.6, 4.5, 8.2, 18.6, 36.6, 27.0, 0.04946, 0, 0, 0, 0, 0, 0 total, 267959, 11157, 11157, 11157, 2.8, 2.6, 4.6, 9.2, 20.9, 69.1, 28.0, 0.04784, 0, 0, 0, 0, 0, 0 total, 279085, 11126, 11126, 11126, 2.8, 2.6, 4.6, 8.2, 16.0, 30.8, 29.0, 0.04620, 0, 0, 0, 0, 0, 0 total, 289773, 10688, 10688, 10688, 3.0, 2.7, 4.9, 7.9, 19.0, 35.5, 30.0, 0.04463, 0, 0, 0, 0, 0, 0 total, 299929, 10156, 10156, 10156, 3.1, 2.7, 4.7, 10.8, 87.3, 90.8, 31.0, 0.04316, 0, 0, 0, 0, 0, 0 total, 310716, 10787, 10787, 10787, 2.9, 2.7, 5.0, 8.9, 16.3, 29.2, 32.0, 0.04179, 0, 0, 0, 0, 0, 0 total, 320931, 10215, 10215, 10215, 2.8, 2.6, 4.5, 7.2, 18.4, 25.5, 33.0, 0.04050, 0, 0, 0, 0, 0, 0 total, 322021, 1090, 1090, 1090, 31.9, 3.1, 11.0, 974.1, 984.6, 985.1, 34.0, 0.15458, 0, 0, 0, 0, 0, 0 total, 333309, 11288, 11288, 11288, 2.8, 2.6, 4.7, 9.9, 21.0, 25.4, 35.0, 0.15035, 0, 0, 0, 0, 0, 0 total, 344239, 10930, 10930, 10930, 2.9, 2.6, 5.0, 9.9, 39.6, 54.8, 36.0, 0.14636, 0, 0, 0, 0, 0, 0 total, 355560, 11321, 11321, 11321, 2.8, 2.6, 4.7, 9.3, 22.8, 41.4, 37.0, 0.14249, 0, 0, 0, 0, 0, 0 total, 364990, 9430, 9430, 9430, 3.4, 2.6, 5.0, 12.3, 141.3, 158.2, 38.0, 0.13902, 0, 0, 0, 0, 0, 0 total, 376297, 11307, 11307, 11307, 2.8, 2.6, 4.5, 8.6, 21.8, 24.0, 39.0, 0.13559, 0, 0, 0, 0, 0, 0 total, 387651, 11354, 11354, 11354, 2.8, 2.5, 4.7, 8.9, 20.3, 26.4, 40.0, 0.13230, 0, 0, 0, 0, 0, 0 total, 394128, 6477, 6477, 6477, 2.7, 2.6, 4.3, 5.8, 12.0, 14.0, 41.0, 0.13099, 0, 0, 0, 0, 0, 0 total, 396263, 2135, 2135, 2135, 21.6, 2.7, 4.7, 1256.2, 1258.3, 1259.3, 42.0, 0.13955, 0, 0, 0, 0, 0, 0 total, 407132, 10869, 10869, 10869, 2.9, 2.6, 4.7, 10.1, 18.8, 24.2, 43.0, 0.13638, 0, 0, 0, 0, 0, 0 total, 418276, 11144, 11144, 11144, 2.8, 2.6, 4.6, 9.0, 15.6, 26.1, 44.0, 0.13327, 0, 0, 0, 0, 0, 0 total, 429779, 11503, 11503, 11503, 2.8, 2.6, 4.3, 5.6, 13.4, 22.0, 45.0, 0.13022, 0, 0, 0, 0, 0, 0 total, 439888, 10109, 10109, 10109, 3.1, 2.6, 4.5, 9.7, 106.7, 108.4, 46.0, 0.12741, 0, 0, 0, 0, 0, 0 total, 450854, 10966, 10966, 10966, 2.9, 2.6, 4.7, 8.5, 21.0, 40.6, 47.0, 0.12470, 0, 0, 0, 0, 0, 0 total, 462404, 11550, 11550, 11550, 2.7, 2.6, 4.3, 5.9, 11.4, 19.3, 48.0, 0.12202, 0, 0, 0, 0, 0, 0 total, 472649, 10245, 10245, 10245, 2.7, 2.5, 4.3, 8.8, 13.8, 43.7, 49.0, 0.11969, 0, 0, 0, 0, 0, 0 total, 476297, 3648, 3648, 3648, 9.8, 2.7, 4.3, 12.1, 804.3, 805.3, 50.0, 0.11649, 0, 0, 0, 0, 0, 0 total, 487422, 11125, 11125, 11125, 2.8, 2.7, 4.5, 6.5, 16.0, 32.0, 51.0, 0.11421, 0, 0, 0, 0, 0, 0 total, 498707, 11285, 11285, 11285, 2.8, 2.6, 4.4, 8.9, 22.1, 32.6, 52.0, 0.11200, 0, 0, 0, 0, 0, 0 total, 509903, 11196, 11196, 11196, 2.8, 2.6, 4.5, 9.0, 16.5, 33.3, 53.0, 0.10988, 0, 0, 0, 0, 0, 0
