[
https://issues.apache.org/jira/browse/HADOOP-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-9361:
-----------------------------------
Status: Open (was: Patch Available)
looking at the warnings. I think that test failure is a false alarm, but will
resubmit to see; I've also tuned s3n's exception logic a bit -that's a bit of a
troublespot.
Andrew, most of this is documentation -what HDFS does and where other
filesystems differ within their "correct" operation, fixes for the other
filesystems to make them more consistent with HDFS and more robust -especially
to {{seek()}} and {{close()}}. For HDFS, all that changes is that out of range
seeks are raised as {{EOFException}} instead of simple {{IOException}} - with
the text unchanged in case people were looking for that in their code.
The thing I'd like input on is actually that specification. Does it make sense
-and can you see obvious mistakes in it. Rename() is the troublespot,
incidentally -in the spec, the implementation and the tests. As for the tests,
they've got a more contract-driven architecture, with an XML file for each FS
defining behaviour, e.g
[hdfs.xml|https://github.com/steveloughran/hadoop-trunk/blob/stevel/HADOOP-9361-filesystem-contract/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/contract/hdfs.xml].
These are in the test JARs to stop them being used by code, though I'd like
them (somehow) to get down to any functional tests in Bigtop.
Have a look at [that
specification|https://github.com/steveloughran/hadoop-trunk/tree/stevel/HADOOP-9361-filesystem-contract/hadoop-common-project/hadoop-common/src/site/markdown/filesystem]
and call out troublespots.
I'll be in the US after hadoop summit if you want to talk on it, though JIRA is
where detailed feedback should go for the historical record.
> Strictly define the expected behavior of filesystem APIs and write tests to
> verify compliance
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-9361
> URL: https://issues.apache.org/jira/browse/HADOOP-9361
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, test
> Affects Versions: 2.4.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-9361-001.patch, HADOOP-9361-002.patch,
> HADOOP-9361-003.patch, HADOOP-9361-004.patch, HADOOP-9361-005.patch,
> HADOOP-9361-006.patch, HADOOP-9361-007.patch, HADOOP-9361-008.patch,
> HADOOP-9361-009.patch, HADOOP-9361-011.patch, HADOOP-9361-012.patch,
> HADOOP-9361-013.patch
>
>
> {{FileSystem}} and {{FileContract}} aren't tested rigorously enough -while
> HDFS gets tested downstream, other filesystems, such as blobstore bindings,
> don't.
> The only tests that are common are those of {{FileSystemContractTestBase}},
> which HADOOP-9258 shows is incomplete.
> I propose
> # writing more tests which clarify expected behavior
> # testing operations in the interface being in their own JUnit4 test classes,
> instead of one big test suite.
> # Having each FS declare via a properties file what behaviors they offer,
> such as atomic-rename, atomic-delete, umask, immediate-consistency -test
> methods can downgrade to skipped test cases if a feature is missing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
