Hello, I had a question about writing a Yarn application. Is it possible to transfer data from the yarn client to the application master and each subsequent container? Basically I want to be able to start my application with some private data on an edge node and have the data copied to the AM and any container it launches using a secure method.
One thing we're considering is using LocalResource, which seems to deviate for security reasons if the LocalResourceVisibility is set to private and the NodeManager is set to secure mode. The article also mentions: "Each PublicLocalizer uses credentials passed as part of ContainerLaunchContext to securely copy the resources from the remote file-system." [1] However, I'm wondering how secure is this method of transferring data? Can I trust that any data I send marked as private is guaranteed only to be accessible by the user of my application? Any other recommendations for how I might accomplish this goal? At the moment, the data is being manually copied to each node and protected with posix permissions, but would like to improve this process. Best, Tim [1] http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/
