Hi Frank
Is map output compression enabled?
The config param would be like
mapred.map.output.compress=true
(It is from my memory, Please cross check)
------Original Message------
From: Frank Grimes
To: [email protected]
ReplyTo: [email protected]
Subject: Can spill to disk be in compressed format to reduce I/O?
Sent: Jan 12, 2012 21:10
Hi All,
We're trying to speed up an M/R job which combines multiple .avro files.
We've noticed that when it spills to disk, it's in uncompressed format.
Is there a way to make it spill temporary segments as .avro with Deflate
compression?
Thanks,
Frank Grimes
Regards
Bejoy K S
