steveloughran commented on PR #4352: URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1159135030
we need a single IOStatisticsContext for all FS instances, so that a task reading from one fs and writing to another would have the stats updated from both actions. And it needs to be in hadoop-common, just like the common audit context, so that code can compile and link against it even without having hadoop-aws or hadoop-azure on the classpath. Making the context a weak ref map ensures that GCs will trigger cleanup. This would lose stats, but not while any stream was active, *or if some code picked up a reference to the thread stats before executing work. That is what I plan to do in spark; we will grab that ref before starting the work, and after it is finished, take a snapshot of it. Oh, and reset the values before work starts -we need that context there too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
