[
https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14488:
------------------------------------
Attachment: output.txt
Full log of the test run. Core operation is the rename:
{code}
Renaming
2017-06-08 13:56:14,285 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:innerRename(850)) - Rename path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
to
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work
2017-06-08 13:56:14,285 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1899)) - Getting path
status for
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
(cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00)
2017-06-08 13:56:14,286 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:get(359)) - Get from
table hwdev-steve-new in region us-west-1:
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
2017-06-08 13:56:14,463 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:get(369)) - Get from
table hwdev-steve-new in region us-west-1 returning for
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00:
PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00;
isDirectory=false; length=10; replication=1; blocksize=1048576;
modification_time=1496926572647; access_time=0; owner=stevel; group=stevel;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false}
2017-06-08 13:56:14,464 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:innerGetFileStatus(1899)) - Getting path
status for
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work
(cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work)
2017-06-08 13:56:14,464 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:get(359)) - Get from
table hwdev-steve-new in region us-west-1:
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work
2017-06-08 13:56:14,641 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:get(369)) - Get from
table hwdev-steve-new in region us-west-1 returning for
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work:
PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work;
isDirectory=true; modification_time=1496926574641; access_time=0;
owner=stevel; group=stevel; permission=rwxrwxrwx; isSymlink=false;
hasAcl=false; isEncrypted=false; isErasureCoded=false};
isEmptyDirectory=UNKNOWN; isDeleted=false}
2017-06-08 13:56:14,819 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:innerRename(934)) - rename: renaming file
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
to
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work
2017-06-08 13:56:14,820 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:copyFile(2269)) - copyFile
cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00 ->
cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/part-00
2017-06-08 13:56:15,424 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1524)) - Delete path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
- recursive false
2017-06-08 13:56:15,424 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:innerDelete(1582)) - delete: Path is a
file
2017-06-08 13:56:15,622 [ScalaTest-main-running-S3AConsistencySuite] INFO
s3a.InconsistentAmazonS3Client
(InconsistentAmazonS3Client.java:isKeyDelayed(200)) - delaying
cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
2017-06-08 13:56:15,819 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:innerDelete(303)) -
Deleting from table hwdev-steve-new in region us-west-1:
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
2017-06-08 13:56:16,003 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:move(446)) - Moving
paths of table hwdev-steve-new in region us-west-1: 1 paths to delete and 1
paths to create
2017-06-08 13:56:16,003 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:move(473)) - move:
auto-create ancestor path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work
for child path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/part-00
2017-06-08 13:56:16,003 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:move(473)) - move:
auto-create ancestor path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite
for child path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/part-00
2017-06-08 13:56:16,003 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:move(473)) - move:
auto-create ancestor path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME for child path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/part-00
2017-06-08 13:56:16,003 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:move(473)) - move:
auto-create ancestor path s3a://hwdev-steve-new/cloud-integration for child
path
s3a://hwdev-steve-new/cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/part-00
2017-06-08 13:56:17,544 [ScalaTest-main-running-S3AConsistencySuite] INFO
s3a.InconsistentAmazonS3Client
(InconsistentAmazonS3Client.java:isKeyDelayed(200)) - delaying
cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
2017-06-08 13:56:17,545 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2018)) - Found path as
directory (with /): 1/0
2017-06-08 13:56:17,545 [ScalaTest-main-running-S3AConsistencySuite] DEBUG
s3a.S3AFileSystem (S3AFileSystem.java:s3GetFileStatus(2025)) - Prefix:
cloud-integration/DELAY_LISTING_ME/S3AConsistencySuite/work/task00/part-00
2017-06-08 13:56:17,547 [ScalaTest-main-running-S3AConsistencySuite] INFO
s3.S3AConsistencySuite (Logging.scala:logInfo(54)) -
{code}
I think somehow the combination of inconsistent remote listing and the rename
operation is causing the DB to get into a mess, or to list itself wrongly. Is
there a way to view it?
> s3guard listStatus fails after renaming file into directory
> -----------------------------------------------------------
>
> Key: HADOOP-14488
> URL: https://issues.apache.org/jira/browse/HADOOP-14488
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
> Priority: Blocker
> Attachments: HADOOP-14488-HADOOP-13345-001.patch, output.txt
>
>
> Running scala integration test with inconsistent s3 client & local DDB enabled
> {code}
> fs.rename("work/task-00/part-00", work)
> fs.listStatus(work)
> {code}
> The list status work fails with a message about the childStatus not being a
> child of the parent.
> Hypothesis: rename isn't updating the child path entry
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
