[
https://issues.apache.org/jira/browse/SOLR-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159595#comment-17159595
]
Erick Erickson commented on SOLR-14653:
---------------------------------------
After looking this over, I decided to just go with some simple string
manipulation in a utility class. ZK paths have nothing really to do with URIs,
Files, Paths, etc. they're just simple strings with no sensitivity to the file
system etc. I toyed a bit with a utility method that would take a bunch of
fragments and assemble them into a Znode path, but then realized that was a
whole can of worms. There are lots and lots of places where we construct a path
just by putting together strings, look at calls to SolrZkClient.create for
instance.
So this patch takes a minimalist approach. There are three cases where
hadoop.fs.Path was brought in outside of something having hdfs in the class
name:
* to get the parent from a string. These now call
ZkMaintenanceUtils.getZkParent.
* to pull in Path.SEPARATOR to construct a path. This is silly. These now just
hard-code "/" like we do everywhere else.
* SolrSnapshotsTool and DiskChecker (a test) which I left alone.
If we're going to do something more robust when it comes to creating ZK paths,
it'll be a much larger undertaking to retrofit all the places we construct them
inline now. I think this address the immediate concern.
All tests pass, I'll commit tomorrow baring objections.
> Replace usages of org.apache.hadoop.fs.Path that are used to construct
> Zookeeper paths
> --------------------------------------------------------------------------------------
>
> Key: SOLR-14653
> URL: https://issues.apache.org/jira/browse/SOLR-14653
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
> Attachments: SOLR-14653.patch
>
>
> See SOLR-13939. We construct Zookeeper paths at least in
> ShardLeaderElectionContextBase.java by using methods in
> org.apache.hadoop.fs.Path e.g. "String parent = new
> Path(leaderPath).getParent().toString()". This works becauseĀ the Hadoop
> method uses forward slashes even on Windows machines.
> If those calls are changed to use nio Paths.get or similar, we can't run on
> Windows. Relying on hdfs classes to construct these paths in code unrelated
> to HDFS is suspicious at best.
> We should examine all inclusions of org.apache.hadoop.fs.Path and replace
> them with something else. Uwe suggests java.net.URI as one possibility.
> We should check to see what Zookeeper provides.
> There are 5 files that import org.apache.hadoop... that do _not_ have HDFS in
> their name, and 4 of those are probably legitimate (SolrSnapshotsTool for
> instance) so checking all of them won't take long.
> From Uwe's comment in SOLR-13939:
> {quote}"Hi,
> Windows is happy again. So your fix helped. I responded to mailing list,
> here again:
> In Zookeeper all paths are forward slashes (as it is a "path" inside the
> zookeeper environment). If you replace that with a NIO Windows Path it
> creates backslashes. This makes Zookeeper fail (of course).
> Actually using the Hadoop class is broken workaround, it may be better to use
> the more abstract java.net.URI class for correct "path" transformations. The
> risk with java.nio.file.Path is also that if the path exists in file system
> it handles the resolving different. So if some user has for some reason a
> local path exactly looking like a zookeper node path it would use different
> behaviour.
> So my suggestion: Use java.net.URI to build the path and use URI.getPath() or
> similar to make a String out of it. The alternative to replace all
> backslashes in the path to forwards slashes is a bad idea for the reasons I
> explained before.
> Now we should replace the hadoop class IMHO by java.net.URI.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]