[
https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044468#comment-16044468
]
Masatake Iwasaki commented on HADOOP-14424:
-------------------------------------------
Thanks for working on this, [~GeLiXin].
It would be nice if duplicate code in Crc32PerformanceTest and
Crc32CPerformanceTest. I think it is ok to just add test cases for CRC32C to
existing Crc32PerformanceTest. In that case, displaying diffs to all of
previous benchmark results could be noisy. It should be omitted or just print
diff to one of the results.
{noformat}
//compare result with previous
for(BenchResult p : previous) {
final double diff = (result.mbps - p.mbps) / p.mbps * 100;
printCell(String.format("%5.1f%%", diff), diffStr.length(), out);
}
previous.add(result);
}
{noformat}
> Add CRC32C performance test.
> ----------------------------
>
> Key: HADOOP-14424
> URL: https://issues.apache.org/jira/browse/HADOOP-14424
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
> Affects Versions: 3.0.0-alpha2
> Reporter: LiXin Ge
> Assignee: LiXin Ge
> Priority: Minor
> Labels: test
> Attachments: HADOOP-14424.patch
>
>
> The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new
> test to compare Crc32C chunked verification implementations.
> This test is based on Crc32PerformanceTest, what I have done in this test is:
> 1.CRC32C performance test.
> 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it
> from this test.
> 3.User can choose either direct buffer or non-directly buffer to run this
> test manually.
> 4.Take use of verifyChunkedSumsByteArray for native to support non-directly
> native test.
> The test result in my environment is:
> [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
> java.version = 1.8.0_111
> java.runtime.name = Java(TM) SE Runtime Environment
> java.runtime.version = 1.8.0_111-b14
> java.vm.version = 25.111-b14
> java.vm.vendor = Oracle Corporation
> java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> java.vm.specification.version = 1.8
> java.specification.version = 1.8
> os.arch = amd64
> os.name = Linux
> os.version = 2.6.33.20
> Data Length = 64 MB
> Trials = 5
> Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
> | bpc | #T || PureJava || Native | % diff |
> | 32 | 1 | 394.0 | 4156.2 | 954.9% |
> | 32 | 2 | 400.5 | 3679.7 | 818.7% |
> | 32 | 4 | 401.8 | 2657.0 | 561.3% |
> | 32 | 8 | 389.1 | 1633.8 | 319.9% |
> | 32 | 16 | 222.2 | 1116.3 | 402.5% |
> | bpc | #T || PureJava || Native | % diff |
> | 64 | 1 | 465.0 | 5931.0 | 1175.5% |
> | 64 | 2 | 468.8 | 1839.2 | 292.3% |
> | 64 | 4 | 460.4 | 2968.3 | 544.7% |
> | 64 | 8 | 452.4 | 1925.7 | 325.6% |
> | 64 | 16 | 246.9 | 1291.8 | 423.3% |
> | bpc | #T || PureJava || Native | % diff |
> | 128 | 1 | 522.0 | 6147.8 | 1077.6% |
> | 128 | 2 | 366.0 | 4758.5 | 1200.2% |
> | 128 | 4 | 307.8 | 3265.1 | 960.8% |
> | 128 | 8 | 283.6 | 2092.2 | 637.6% |
> | 128 | 16 | 219.9 | 1226.1 | 457.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 256 | 1 | 550.7 | 3177.6 | 477.0% |
> | 256 | 2 | 538.6 | 1933.2 | 258.9% |
> | 256 | 4 | 427.2 | 3278.1 | 667.3% |
> | 256 | 8 | 420.8 | 2272.3 | 440.0% |
> | 256 | 16 | 294.0 | 1311.2 | 346.0% |
> | bpc | #T || PureJava || Native | % diff |
> | 512 | 1 | 553.4 | 3690.4 | 566.9% |
> | 512 | 2 | 455.6 | 4974.1 | 991.7% |
> | 512 | 4 | 494.2 | 3406.4 | 589.2% |
> | 512 | 8 | 431.4 | 2257.0 | 423.2% |
> | 512 | 16 | 316.3 | 1272.0 | 302.2% |
> | bpc | #T || PureJava || Native | % diff |
> | 1024 | 1 | 566.1 | 3520.0 | 521.8% |
> | 1024 | 2 | 508.7 | 4437.4 | 772.3% |
> | 1024 | 4 | 520.7 | 3422.6 | 557.4% |
> | 1024 | 8 | 501.8 | 2124.7 | 323.4% |
> | 1024 | 16 | 340.6 | 1305.0 | 283.2% |
> | bpc | #T || PureJava || Native | % diff |
> | 2048 | 1 | 535.1 | 5438.5 | 916.4% |
> | 2048 | 2 | 537.3 | 4668.3 | 768.8% |
> | 2048 | 4 | 529.2 | 2417.2 | 356.7% |
> | 2048 | 8 | 485.1 | 2249.8 | 363.8% |
> | 2048 | 16 | 334.3 | 1265.7 | 278.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 4096 | 1 | 563.0 | 7264.0 | 1190.1% |
> | 4096 | 2 | 538.8 | 5681.4 | 954.4% |
> | 4096 | 4 | 528.9 | 3107.6 | 487.5% |
> | 4096 | 8 | 521.8 | 2168.9 | 315.6% |
> | 4096 | 16 | 336.5 | 1256.5 | 273.4% |
> | bpc | #T || PureJava || Native | % diff |
> | 8192 | 1 | 521.1 | 6842.4 | 1213.1% |
> | 8192 | 2 | 543.0 | 5510.2 | 914.8% |
> | 8192 | 4 | 526.1 | 3759.5 | 614.6% |
> | 8192 | 8 | 480.1 | 2254.1 | 369.6% |
> | 8192 | 16 | 336.4 | 1213.3 | 260.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 16384 | 1 | 521.7 | 3695.7 | 608.3% |
> | 16384 | 2 | 540.8 | 5610.5 | 937.4% |
> | 16384 | 4 | 539.1 | 3313.0 | 514.6% |
> | 16384 | 8 | 518.7 | 2198.4 | 323.8% |
> | 16384 | 16 | 337.2 | 1237.8 | 267.1% |
> | bpc | #T || PureJava || Native | % diff |
> | 32768 | 1 | 532.1 | 3696.0 | 594.6% |
> | 32768 | 2 | 541.4 | 4647.4 | 758.4% |
> | 32768 | 4 | 536.6 | 2325.1 | 333.3% |
> | 32768 | 8 | 518.8 | 2036.9 | 292.6% |
> | 32768 | 16 | 337.8 | 1232.0 | 264.7% |
> | bpc | #T || PureJava || Native | % diff |
> | 65536 | 1 | 527.7 | 8240.0 | 1461.4% |
> | 65536 | 2 | 561.9 | 5472.3 | 873.9% |
> | 65536 | 4 | 536.5 | 3725.2 | 594.4% |
> | 65536 | 8 | 482.8 | 2040.2 | 322.5% |
> | 65536 | 16 | 335.9 | 1240.4 | 269.3% |
> Elapsed 77.4s
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
