Bernd, Regarding the Hadoop version for VFS 2.1, why not use the latest on the first release of the HDFS provider? The Hadoop 1.1.2 release was released in Feb 2013.
I just built 2.1-SNAPSHOT over the holidays with JDK 6, 7, and 8 on Ubuntu. What type of test errors are you getting? Testing is disabled on Windows unless you decide to pull in windows artifacts attached to VFS-530. However, those artifacts are associated with patch 3 and are for Hadoop 2.4.0. Updating to 2.4.0 would also be sufficient in my opinion. I started up Hadoop 2.6.0 on my laptop, created a directory and file, then used the VFS shell to list and view the contents (remember, HDFS provider is read-only currently). Here is the what I did: ./hadoop fs -ls / ./hadoop fs -mkdir /vfs-test ./hadoop fs -ls / Found 1 items drwxr-xr-x - dave supergroup 0 2015-01-09 21:50 /vfs-test echo "This is a test" > /tmp/test.txt ./hadoop fs -copyFromLocal /tmp/test.txt /vfs-test/text.txt ./hadoop fs -ls -R / drwxr-xr-x - dave supergroup 0 2015-01-09 21:56 /vfs-test -rw-r--r-- 3 dave supergroup 15 2015-01-09 21:56 /vfs-test/text.txt ./hadoop fs -cat /vfs-test/text.txt This is a test unset REP unset HADOOP_HOME unset HADOOP_CLASSPATH unset LIBS REP=~/.m2/repository HADOOP_HOME=/home/dave/Software/hadoop-2.6.0 HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath` LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar LIBS=$LIBS:$HADOOP_CLASSPATH java -cp $LIBS org.apache.commons.vfs2.example.Shell 10:01:41 ~/EclipseWorkspace/commons-vfs2-project$ unset REP 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_HOME 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ unset HADOOP_CLASSPATH 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ unset LIBS 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ REP=~/.m2/repository 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_HOME=/home/dave/Software/hadoop-2.6.0 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ HADOOP_CLASSPATH=`$HADOOP_HOME/bin/hadoop classpath` 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$REP/commons-logging/commons-logging/1.2/commons-logging-1.2.jar 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:core/target/commons-vfs2-2.1-SNAPSHOT.jar:examples/target/commons-vfs2-examples-2.1-SNAPSHOT.jar:sandbox/target/commons-VFS2-sandbox-2.1.jar 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ LIBS=$LIBS:$HADOOP_CLASSPATH 10:01:43 ~/EclipseWorkspace/commons-vfs2-project$ java -cp $LIBS org.apache.commons.vfs2.example.Shell 15/01/09 22:01:44 INFO impl.StandardFileSystemManager: Using "/tmp/vfs_cache" as temporary files store. VFS Shell 2.1-SNAPSHOT > info Default manager: "org.apache.commons.vfs2.impl.StandardFileSystemManager" version 2.1-SNAPSHOT Provider Schemes: [https, res, gz, hdfs, sftp, ftps, ram, http, file, ftp, tmp, bz2] Virtual Schemes: [zip, war, par, ear, jar, sar, ejb3, tar, tbz2, tgz] > info hdfs Provider Info for scheme "hdfs": capabilities: [GET_TYPE, READ_CONTENT, URI, GET_LAST_MODIFIED, ATTRIBUTES, RANDOM_ACCESS_READ, DIRECTORY_READ_CONTENT, LIST_CHILDREN] > ls hdfs://Dave-laptop:8020/ 15/01/09 22:02:06 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS 15/01/09 22:02:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Contents of hdfs://dave-laptop:8020/ vfs-test/ > ls hdfs://Dave-laptop:8020/vfs-test/ Contents of hdfs://dave-laptop:8020/vfs-test text.txt > cat hdfs://Dave-laptop:8020/vfs-test/text.txt This is a test ----- Original Message ----- From: "Bernd Eckenfels" <e...@zusammenkunft.net> To: dlmar...@comcast.net Cc: "Commons Developers List" <dev@commons.apache.org> Sent: Friday, January 9, 2015 8:13:27 PM Subject: Re: [VFS] Implementing custom hdfs file system using commons-vfs 2.0 Am Sat, 10 Jan 2015 01:04:56 +0000 (UTC) schrieb dlmar...@comcast.net: > To answer your question about the HDFS version, I went back and > looked at the patches I supplied for VFS-530. In the patches that I > supplied to bump the Hadoop version to 2.4.0 and 2.6.0 the only > changes were in the pom. This should mean that, for the parts of the > Hadoop client objects that I am using, they are API compatible. Well, you never know. But if thats the case there is also no problem if we stay at the older version for now. > For the example, the list of jars depends on the Hadoop version. I > think one way to handle that is to use the `hadoop classpath` command > to create the classpath. The example would then look like: Looks great, thanks. > VFS Shell 2.1-SNAPSHOT > > info > Default manager: > "org.apache.commons.vfs2.impl.StandardFileSystemManager" version > 2.1-SNAPSHOT Provider Schemes: [https, res, gz, hdfs, sftp, ftps, > ram, http, file, ftp, tmp, bz2] Virtual Schemes: [zip, war, par, ear, > jar, sar, ejb3, tar, tbz2, tgz] > > > Is this sufficient? Did you try to use ls or another command to actually connect to a hdfs name node? I think you should be able to provide "user:password@host" with the Shell. I have currently the problem, that the hdfs tests fail (under Linux) with some form of directory lock problem. I am not sure if this worked before (as I havent tested on my windows). I will see if the version upgrade helps, if yes I will for sure include it in 2.1 Greetings Bernd > > Dave > > ----- Original Message ----- > > From: "Bernd Eckenfels" <e...@zusammenkunft.net> > To: dlmar...@comcast.net > Cc: "Commons Developers List" <dev@commons.apache.org> > Sent: Friday, January 9, 2015 9:05:55 AM > Subject: Re: [VFS] Implementing custom hdfs file system using > commons-vfs 2.0 > > Hello Dave, > > for the download page (staged: > http://people.apache.org/~ecki/commons-vfs/download.html) I need a > list of libraries needed to run VFS to access hdfs. Could you maybe > produce an example VFS Shell session similiar to > > https://wiki.apache.org/commons/VfsReleaseState > > where you list the command line needed. It would be best if this is > an public HDFS instance (but I guess there is no such thing?) > > BTW: I had asked in the VFS-530 bug about how commonplace the > different hdfs APIs are, and if it is really good if we bump the > minimum version. As I understand it. Will HDFS with 1.1.2 be able to > communicate with 2.6 instances? If yes I would prefer to keep it at > 1.2 in this release. WDYT? > > Gruss > Bernd > >
