steveloughran commented on pull request #2069:
URL: https://github.com/apache/hadoop/pull/2069#issuecomment-680292512
...no, I haven't reviewed anyone elses work today, just been rebasing and
wiring up the marker and async listing changes.
To put them both together, the "s3guard markers" audit tool, will for a
verbose run, print the stats from the listing
```
bin/hadoop s3guard markers -audit -limit 20000 -verbose s3a://landsat-pds/
The directory marker policy of s3a://landsat-pds is "Delete"
2020-08-25 22:57:56,738 [main] INFO tools.MarkerTool
(DurationInfo.java:<init>(77)) - Starting: marker scan s3a://landsat-pds/
Scanned 1,000 objects
Scanned 2,000 objects
Scanned 3,000 objects
Scanned 4,000 objects
Scanned 5,000 objects
Scanned 6,000 objects
Scanned 7,000 objects
Scanned 8,000 objects
Scanned 9,000 objects
Scanned 10,000 objects
Scanned 11,000 objects
Scanned 12,000 objects
Scanned 13,000 objects
Scanned 14,000 objects
Scanned 15,000 objects
Scanned 16,000 objects
Scanned 17,000 objects
Scanned 18,000 objects
Scanned 19,000 objects
Scanned 20,000 objects
Limit of scan reached - 20,000 objects
Listing statistics:
counters=((object_continue_list_request=20) (object_list_request=1));
gauges=(); minimums=((object_continue_list_request.min=441)
(object_list_request.min=1375)); maximums=((object_list_request.max=1375)
(object_continue_list_request.max=790));
means=((object_list_request.mean=(sum=1375, samples=1, mean=1375.0000))
(object_continue_list_request.mean=(sum=10145, samples=19, mean=533.9474)));
2020-08-25 22:58:08,303 [main] INFO tools.MarkerTool
(DurationInfo.java:close(98)) - marker scan s3a://landsat-pds/: duration
0:11.566s
Listed 20000 objects under s3a://landsat-pds/
No surplus directory markers were found under s3a://landsat-pds/
Listing limit reached before completing the scan
Storage Statistics for s3a://landsat-pds
op_get_file_status 1
object_list_request 2
object_continue_list_request 20
2020-08-25 22:58:08,312 [main] INFO util.ExitUtil
(ExitUtil.java:terminate(210)) - Exiting with status 3:
```
Even for this operation we are saving a couple of seconds: the sum of all
the list operations are 13s, but the wall time is only 11 seconds. That's two
seconds of my life saved :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
