[
https://issues.apache.org/jira/browse/HADOOP-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13736368#comment-13736368
]
Yu Li commented on HADOOP-9863:
-------------------------------
According to our test on hadoop snappy on ppc64 platform, length of generated
.snappy file is almost empty. After days of investigation we finally located
the problem and below are the analysis:
size_t is defined as "unsigned long int" on 64 bit platform, and jint is
defined as "signed 32bit
int"(http://docs.oracle.com/javase/1.5.0/docs/guide/jni/spec/types.html for the
jni spec), so if we cast size_t type pointer to jint, it will cut-off 32bit
values in the higher adress and only reserve those in the lower address; If the
platform is Big-Endian, and the value stored in the pointer is less than
0xffffffff, then value get through the post-cast pointer will be 0.
Here is a simple program can prove the above analysis:
{code:title=test.c|borderStyle=solid}
#include <stdio.h>
#include <jni.h>
void main(){
jint x;
size_t * p_x = &x;
* p_x = 0x01;
printf("****************************************\n");
printf("Test-1: value less than 0x7fffffff\n");
printf("Before pointer type cast: %ld\n",*p_x);
printf("After pointer type cast: %ld\n",x);
printf("After value type cast: %ld\n",(jint)*p_x);
printf("****************************************\n");
* p_x = 0x80000000;
printf("Test-2: value larger than 0x7fffffff\n");
printf("Before pointer type cast: %ld\n",*p_x);
printf("After pointer type cast: %ld\n",x);
printf("After value type cast: %ld\n",(jint)*p_x);
printf("****************************************\n");
* p_x = 0x1ffffffff;
printf("Test-3: value larger than 0xffffffff\n");
printf("Before pointer type cast: %ld\n",*p_x);
printf("After pointer type cast: %ld\n",x);
printf("****************************************\n");
}
{code}
And here is the test result:
{noformat}
****************************************
Test-1: value less than 0x7fffffff
Before pointer type cast: 1
After pointer type cast: 0
After value type cast: 1
****************************************
Test-2: value larger than 0x7fffffff
Before pointer type cast: 2147483648
After pointer type cast: 0
After value type cast: -2147483648
****************************************
Test-3: value larger than 0xffffffff
Before pointer type cast: 8589934591
After pointer type cast: 1
****************************************
{noformat}
Test-1 proves the data loss caused by pointer type cast (as we observed in
snappy of branch-1 hadoop), Test-2 proves the necessarity to set JINT_MAX as
done in HADOOP-8686, Test-3 proves the higher 32 bit(store in lower address as
big-endian) will be preserved with lower 32 bit cut-off
The patch attached probably can help better understand this issue.
> Snappy compression cannot work on Big-Endian 64 bit platform w/o backporting
> HADOOP-8686
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-9863
> URL: https://issues.apache.org/jira/browse/HADOOP-9863
> Project: Hadoop Common
> Issue Type: Bug
> Components: native
> Affects Versions: 1.1.1, 1.1.2, 1.2.1
> Reporter: Yu Li
> Assignee: Yu Li
> Labels: native, ppc64, snappy
>
> w/o changes made in HADOOP-8686 on SnappyCompressor.c, snappy compression in
> branch-1 hadoop on big endian 64 bit platform (ppc64 for example) will
> generate incorrect .snappy (almost-empty) file because of type casting from
> size_t to jint. Will include more detailed analysis in comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
