Repository: accumulo Updated Branches: refs/heads/asf-site 45d81dfc4 -> 604500f88 refs/heads/gh-pages 6538290f3 -> f30cc51b1
Add a paragraph on the smaller keys for index blocks improvement Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/f30cc51b Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/f30cc51b Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/f30cc51b Branch: refs/heads/gh-pages Commit: f30cc51b11c0c2c0289f19579ca78a98753e547f Parents: 6538290 Author: Josh Elser <els...@apache.org> Authored: Thu Jun 23 11:11:05 2016 -0400 Committer: Josh Elser <els...@apache.org> Committed: Thu Jun 23 11:11:05 2016 -0400 ---------------------------------------------------------------------- release_notes/1.7.2.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/accumulo/blob/f30cc51b/release_notes/1.7.2.md ---------------------------------------------------------------------- diff --git a/release_notes/1.7.2.md b/release_notes/1.7.2.md index eebdc70..dccb092 100644 --- a/release_notes/1.7.2.md +++ b/release_notes/1.7.2.md @@ -31,10 +31,14 @@ There was a bug ([ACCUMULO-4148][ACCUMULO-4148]) where multiple put calls with i An improvement was introduced to allow a max age before WAL files would be automatically rolled. Without a max age, they could stay open for writing indefinitely, blocking the Hadoop DataNode decomissioning process. For more information, see [ACCUMULO-4004][ACCUMULO-4004]. -### Remove copy of cached RFile index blocks +### Remove unnecessary copy of cached RFile index blocks Accumulo maintains an cache for file blocks in-memory as a performance optimization. This can be done safely because Accumulo RFiles are immutable, thus their blocks are also immutable. There are two types of these blocks: index and data blocks. Index blocks refer to the b-tree style index inside of each Accumulo RFile, while data blocks contain the sorted Key-Value pairs. In previous versions, when Accumulo extracted an Index block from the in-memory cache, it would copy the data. [ACCUMULO-4164][ACCUMULO-4164] removes this unnecessary copy as the contents are immutable and can be passed by reference. Ensuring that the Index blocks are not copied when accessed from the cache is a big performance gain at the file-access level. +### Analyze Key-length to avoid choosing large Keys for RFile Index blocks + +Accumulo's RFile index blocks are made up of a Key which exists in the file and points to that specific location in the corresponding RFile data block. Thus, the size of the RFile index blocks is largely dominated by the size of the Keys which are used by the index. [ACCUMULO-4314][ACCUMULO-4314] is an improvement which uses statistics on the length of the Keys in the Rfile to avoid choosing Keys for the index whose length is greater than three standard deviations for the RFile. By choosing smaller Keys for the index, Accumulo can access the RFile index faster and keep more Index blocks cached in memory. Initial tests showed that with this change, the RFile index size was nearly cut in half. + ### Minor performance improvements. Tablet servers would previously always hsync at the start of a minor compaction, causing delays in the write pipeline. These additional syncs were determined to provide no additional durability guarantees and have been removed. See [ACCUMULO-4112][ACCUMULO-4112] for additional detail. @@ -83,3 +87,4 @@ HDFS High-Availability instances, forcing NameNode failover. [ACCUMULO-4173]: https://issues.apache.org/jira/browse/ACCUMULO-4173 [ACCUMULO-4151]: https://issues.apache.org/jira/browse/ACCUMULO-4151 [ACCUMULO-4164]: https://issues.apache.org/jira/browse/ACCUMULO-4164 +[ACCUMULO-4314]: https://issues.apache.org/jira/browse/ACCUMULO-4314
