Looking at the code all of the instances of Counter for the increment method just to a value += incr. They are not going to be much more expensive then keeping the counter yourself internally. What happens in that at periodic intervals the counters are then sent to the JT (pre YARN) or the ApplicationMaster (in YARN) where it keeps track of them. It should really have no impact on how often you increment them.
--Bobby Evans On 12/8/11 12:48 PM, "Steve Lewis" <[email protected]> wrote: I have several counters that I maintain to allow me to keep statistics on critical operations. I have my code incrementing the counters in an inner loop partly to make sure my job is not killed for not making progress. It would be very easy to keep an internal counter and increment the Hadoop value less frequently. Assuming I am currently incrementing a counter several million times in a reduce task- is this costing me performance and would I be better off incrementing less frequently
