[
https://issues.apache.org/jira/browse/HADOOP-9295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611604#comment-13611604
]
David Parks commented on HADOOP-9295:
-------------------------------------
specifically, add a call to clear:
{code}
/*
* TODO BUG FIX
*/
classToIdMap.clear();
idToClassMap.clear();
{code}
and remove the pre-cached class types
{code}
/*
* TODO BUG FIX
*/
// addToMap(ArrayWritable.class,
// Byte.valueOf(Integer.valueOf(-127).byteValue()));
// addToMap(BooleanWritable.class,
// Byte.valueOf(Integer.valueOf(-126).byteValue()));
// addToMap(BytesWritable.class,
// Byte.valueOf(Integer.valueOf(-125).byteValue()));
// addToMap(FloatWritable.class,
// Byte.valueOf(Integer.valueOf(-124).byteValue()));
// addToMap(IntWritable.class,
// Byte.valueOf(Integer.valueOf(-123).byteValue()));
// addToMap(LongWritable.class,
// Byte.valueOf(Integer.valueOf(-122).byteValue()));
// addToMap(MapWritable.class,
// Byte.valueOf(Integer.valueOf(-121).byteValue()));
// addToMap(MD5Hash.class,
// Byte.valueOf(Integer.valueOf(-120).byteValue()));
// addToMap(NullWritable.class,
// Byte.valueOf(Integer.valueOf(-119).byteValue()));
// addToMap(ObjectWritable.class,
// Byte.valueOf(Integer.valueOf(-118).byteValue()));
// addToMap(SortedMapWritable.class,
// Byte.valueOf(Integer.valueOf(-117).byteValue()));
// addToMap(Text.class,
// Byte.valueOf(Integer.valueOf(-116).byteValue()));
// addToMap(TwoDArrayWritable.class,
// Byte.valueOf(Integer.valueOf(-115).byteValue()));
//
// // UTF8 is deprecated so we don't support it
//
// addToMap(VIntWritable.class,
// Byte.valueOf(Integer.valueOf(-114).byteValue()));
// addToMap(VLongWritable.class,
// Byte.valueOf(Integer.valueOf(-113).byteValue()));
{code}
> AbstractMapWritable throws exception when calling readFields() multiple times
> when the maps contain different class types
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-9295
> URL: https://issues.apache.org/jira/browse/HADOOP-9295
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 1.0.3
> Reporter: David Parks
> Assignee: Karthik Kambatla
> Priority: Critical
> Attachments: MapWritableBugTest.java, test-hadoop-9295.patch
>
>
> Verified the trunk looks the same as 1.0.3 for this issue.
> When mappers output MapWritables with different class types, then they are
> read in on the Reducer via an iterator (multiple calls to readFields without
> instantiating a new object) you'll get this:
> java.lang.IllegalArgumentException: Id 1 exists but maps to
> org.me.ClassTypeOne and not org.me.ClassTypeTwo
> at
> org.apache.hadoop.io.AbstractMapWritable.addToMap(AbstractMapWritable.java:73)
> at
> org.apache.hadoop.io.AbstractMapWritable.readFields(AbstractMapWritable.java:201)
> It happens because AbstractMapWritable accumulates class type entries in its
> ClassType to ID (and vice versa) hashmaps.
> Those accumulating classtype-to-id hashmaps need to be cleared to support
> multiple calls to readFields().
> I've attached a JUnit test that both demonstrates the problem and contains an
> embedded, fixed version of MapWritable and ArrayMapWritable (note the //TODO
> comments in the code where it was fixed in 2 places).
> If there's a better way to submit this recommended bug fix, someone please
> feel free to let me know.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
