We have this, if it is any help
write-4k-seq: (groupid=0, jobs=1): err= 0: pid=1446964: Fri May 24
19:41:48 2019
write: IOPS=760, BW=3042KiB/s (3115kB/s)(535MiB/180001msec)
slat (usec): min=7, max=234, avg=16.59, stdev=13.59
clat (usec): min=786, max=167483, avg=1295.60, stdev=1933.25
lat (usec): min=810, max=167492, avg=1312.46, stdev=1933.67
clat percentiles (usec):
| 1.00th=[ 914], 5.00th=[ 979], 10.00th=[ 1020], 20.00th=[
1074],
| 30.00th=[ 1123], 40.00th=[ 1172], 50.00th=[ 1205], 60.00th=[
1254],
| 70.00th=[ 1319], 80.00th=[ 1401], 90.00th=[ 1516], 95.00th=[
1631],
| 99.00th=[ 2704], 99.50th=[ 3949], 99.90th=[ 5145], 99.95th=[
5538],
| 99.99th=[139461]
bw ( KiB/s): min= 625, max= 3759, per=80.13%, avg=2436.63,
stdev=653.68, samples=359
iops : min= 156, max= 939, avg=608.76, stdev=163.41,
samples=359
lat (usec) : 1000=7.83%
lat (msec) : 2=90.27%, 4=1.42%, 10=0.45%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=0.52%, sys=1.55%, ctx=162087, majf=0, minf=28
IO depths : 1=117.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,136889,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randwrite-4k-seq: (groupid=1, jobs=1): err= 0: pid=1448032: Fri May 24
19:41:48 2019
write: IOPS=655, BW=2620KiB/s (2683kB/s)(461MiB/180001msec)
slat (usec): min=7, max=120, avg=10.79, stdev= 6.22
clat (usec): min=897, max=77251, avg=1512.76, stdev=368.36
lat (usec): min=906, max=77262, avg=1523.77, stdev=368.54
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1205], 10.00th=[ 1254], 20.00th=[
1319],
| 30.00th=[ 1369], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[
1532],
| 70.00th=[ 1598], 80.00th=[ 1663], 90.00th=[ 1778], 95.00th=[
1893],
| 99.00th=[ 2540], 99.50th=[ 2933], 99.90th=[ 3392], 99.95th=[
4080],
| 99.99th=[ 6194]
bw ( KiB/s): min= 1543, max= 2830, per=79.66%, avg=2087.02,
stdev=396.14, samples=359
iops : min= 385, max= 707, avg=521.39, stdev=99.06,
samples=359
lat (usec) : 1000=0.06%
lat (msec) : 2=97.19%, 4=2.70%, 10=0.04%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.39%, sys=1.13%, ctx=118477, majf=0, minf=50
IO depths : 1=116.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=0,117905,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
read-4k-seq: (groupid=2, jobs=1): err= 0: pid=1449103: Fri May 24
19:41:48 2019
read: IOPS=2736, BW=10.7MiB/s (11.2MB/s)(1924MiB/180001msec)
slat (usec): min=6, max=142, avg= 9.26, stdev= 5.02
clat (usec): min=152, max=13253, avg=353.73, stdev=98.92
lat (usec): min=160, max=13262, avg=363.24, stdev=99.15
clat percentiles (usec):
| 1.00th=[ 182], 5.00th=[ 215], 10.00th=[ 239], 20.00th=[
273],
| 30.00th=[ 306], 40.00th=[ 330], 50.00th=[ 355], 60.00th=[
375],
| 70.00th=[ 396], 80.00th=[ 420], 90.00th=[ 461], 95.00th=[
498],
| 99.00th=[ 586], 99.50th=[ 635], 99.90th=[ 775], 99.95th=[
889],
| 99.99th=[ 1958]
bw ( KiB/s): min= 5883, max=13817, per=79.66%, avg=8720.01,
stdev=1895.05, samples=359
iops : min= 1470, max= 3454, avg=2179.63, stdev=473.78,
samples=359
lat (usec) : 250=13.13%, 500=82.11%, 750=4.64%, 1000=0.09%
lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=1.31%, sys=3.69%, ctx=493433, majf=0, minf=32
IO depths : 1=115.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=492640,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randread-4k-seq: (groupid=3, jobs=1): err= 0: pid=1450173: Fri May 24
19:41:48 2019
read: IOPS=1812, BW=7251KiB/s (7425kB/s)(1275MiB/180001msec)
slat (usec): min=6, max=161, avg=10.25, stdev= 6.37
clat (usec): min=182, max=23748, avg=538.35, stdev=136.71
lat (usec): min=189, max=23758, avg=548.86, stdev=137.19
clat percentiles (usec):
| 1.00th=[ 265], 5.00th=[ 310], 10.00th=[ 351], 20.00th=[
445],
| 30.00th=[ 494], 40.00th=[ 519], 50.00th=[ 537], 60.00th=[
562],
| 70.00th=[ 594], 80.00th=[ 644], 90.00th=[ 701], 95.00th=[
742],
| 99.00th=[ 816], 99.50th=[ 840], 99.90th=[ 914], 99.95th=[
1172],
| 99.99th=[ 2442]
bw ( KiB/s): min= 4643, max= 7991, per=79.54%, avg=5767.26,
stdev=1080.89, samples=359
iops : min= 1160, max= 1997, avg=1441.43, stdev=270.23,
samples=359
lat (usec) : 250=0.57%, 500=31.98%, 750=62.92%, 1000=4.46%
lat (msec) : 2=0.05%, 4=0.01%, 10=0.01%, 50=0.01%
cpu : usr=1.07%, sys=2.69%, ctx=327838, majf=0, minf=76
IO depths : 1=116.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=326298,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
rw-4k-seq: (groupid=4, jobs=1): err= 0: pid=1451237: Fri May 24 19:41:48
2019
read: IOPS=557, BW=2228KiB/s (2282kB/s)(392MiB/180002msec)
slat (usec): min=6, max=123, avg=11.27, stdev= 7.67
clat (usec): min=189, max=126636, avg=405.01, stdev=834.36
lat (usec): min=198, max=126646, avg=416.52, stdev=834.50
clat percentiles (usec):
| 1.00th=[ 249], 5.00th=[ 277], 10.00th=[ 297], 20.00th=[
330],
| 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 379], 60.00th=[
396],
| 70.00th=[ 416], 80.00th=[ 445], 90.00th=[ 494], 95.00th=[
537],
| 99.00th=[ 660], 99.50th=[ 758], 99.90th=[ 3851], 99.95th=[
4047],
| 99.99th=[ 9241]
bw ( KiB/s): min= 638, max= 2937, per=79.18%, avg=1764.06,
stdev=471.57, samples=359
iops : min= 159, max= 734, avg=440.69, stdev=117.93,
samples=359
write: IOPS=558, BW=2233KiB/s (2286kB/s)(392MiB/180002msec)
slat (usec): min=6, max=121, avg=11.98, stdev= 9.00
clat (usec): min=812, max=144205, avg=1358.81, stdev=2563.77
lat (usec): min=821, max=144213, avg=1371.03, stdev=2563.99
clat percentiles (usec):
| 1.00th=[ 938], 5.00th=[ 1004], 10.00th=[ 1045], 20.00th=[
1106],
| 30.00th=[ 1172], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[
1303],
| 70.00th=[ 1352], 80.00th=[ 1418], 90.00th=[ 1516], 95.00th=[
1631],
| 99.00th=[ 3392], 99.50th=[ 4359], 99.90th=[ 5145], 99.95th=[
7046],
| 99.99th=[135267]
bw ( KiB/s): min= 644, max= 3014, per=79.26%, avg=1769.09,
stdev=469.77, samples=359
iops : min= 161, max= 753, avg=441.93, stdev=117.47,
samples=359
lat (usec) : 250=0.58%, 500=44.94%, 750=4.17%, 1000=2.40%
lat (msec) : 2=46.57%, 4=0.95%, 10=0.35%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.02%
cpu : usr=0.67%, sys=1.88%, ctx=210045, majf=0, minf=32
IO depths : 1=117.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=100281,100473,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
randrw-4k-seq: (groupid=5, jobs=1): err= 0: pid=1452325: Fri May 24
19:41:48 2019
read: IOPS=435, BW=1743KiB/s (1785kB/s)(306MiB/180001msec)
slat (usec): min=6, max=156, avg=11.39, stdev= 7.87
clat (usec): min=159, max=128737, avg=533.01, stdev=763.72
lat (usec): min=188, max=128747, avg=544.64, stdev=763.95
clat percentiles (usec):
| 1.00th=[ 227], 5.00th=[ 265], 10.00th=[ 322], 20.00th=[
392],
| 30.00th=[ 429], 40.00th=[ 461], 50.00th=[ 502], 60.00th=[
545],
| 70.00th=[ 594], 80.00th=[ 644], 90.00th=[ 701], 95.00th=[
750],
| 99.00th=[ 922], 99.50th=[ 2409], 99.90th=[ 4146], 99.95th=[
4293],
| 99.99th=[16909]
bw ( KiB/s): min= 451, max= 2345, per=80.79%, avg=1408.09,
stdev=457.23, samples=359
iops : min= 112, max= 586, avg=351.64, stdev=114.32,
samples=359
write: IOPS=434, BW=1738KiB/s (1779kB/s)(305MiB/180001msec)
slat (usec): min=6, max=167, avg=12.06, stdev= 9.08
clat (usec): min=958, max=160031, avg=1737.21, stdev=3854.81
lat (usec): min=966, max=160040, avg=1749.51, stdev=3854.97
clat percentiles (usec):
| 1.00th=[ 1156], 5.00th=[ 1254], 10.00th=[ 1303], 20.00th=[
1369],
| 30.00th=[ 1434], 40.00th=[ 1483], 50.00th=[ 1532], 60.00th=[
1598],
| 70.00th=[ 1663], 80.00th=[ 1745], 90.00th=[ 1876], 95.00th=[
2040],
| 99.00th=[ 4490], 99.50th=[ 5014], 99.90th=[ 10683],
99.95th=[132645],
| 99.99th=[141558]
bw ( KiB/s): min= 456, max= 2142, per=80.81%, avg=1403.75,
stdev=446.02, samples=359
iops : min= 114, max= 535, avg=350.53, stdev=111.47,
samples=359
lat (usec) : 250=1.65%, 500=23.36%, 750=22.59%, 1000=2.03%
lat (msec) : 2=47.21%, 4=2.32%, 10=0.79%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.04%
cpu : usr=0.57%, sys=1.48%, ctx=162742, majf=0, minf=54
IO depths : 1=118.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
issued rwt: total=78443,78200,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: woensdag 5 juni 2019 16:53
To: [email protected]
Cc: ceph-users
Subject: *****SPAM***** Re: [ceph-users] Single threaded IOPS on SSD
pool.
> Ok, average network latency from VM to OSD's ~0.4ms.
It's rather bad, you can improve the latency by 0.3ms just by upgrading
the network.
> Single threaded performance ~500-600 IOPS - or average latency of
> 1.6ms Is that comparable to what other are seeing?
Good "reference" numbers are 0.5ms for reads (~2000 iops) and 1ms for
writes (~1000 iops).
I confirm that the most powerful thing to do is disabling CPU powersave
(governor=performance + cpupower -D 0). You usually get 2x single thread
iops at once.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
