Hello, I have a couple of questions regarding mapreduce configurations.
We install various platforms on data nodes that require mixed set of native libraries. Part of the problem is that in general case, this software platforms may be installed into different locations in the backend. (we try to unify it, but still). What it means, it may require site-specific -Djava.library.path setting. I configured individual jvm options (mapred.child.java.opts) on each node to include specific set of paths. However, i encountered 2 problems: #1: my setting doesn't go into effect unless I also declare it final in the data node. It's just being overriden by default -Xmx200 value from the driver EVEN when i don't set it on the driver at all (and there seems to be no way to unset it). However, using "final" spec at the backend creates a problem if some of numerous jobs we run wishes to override the setting still. The ideal behavior is if i don't set it in the driver, then backend value kicks in, otherwise it's driver's value. But i did not find a way to do that for this particular setting for some reason.Could somebody clarify the best workaround? thank you. #2. Ideal behavior would actually be to merge driver-specific and backend-specific settings. E.g. backend may need to configure specific software package locations while client may wish sometimes to set heap etc. Is there a best practice to achieve this effect? Thank you very much in advance. -Dmitriy
